, <style> <body class="inline|sbs"> <header class="cover|top"> <h1>; with CoverPage: file names, settings, hidden categories, completeness line, counts by change type; then the key <main aria-label="Redline"> the layout below <section class="ledger"> "Differences not shown": counts per category, first 50 locations each <section class="notes"> warnings and errors from the stages (only when there are any) <footer> "Compared by OneDiff x.y.z", input names with SHA-256, timestamp if given ``` **Style as data.** The sheet is generated from the `RedlineStyle`: each mark (insertion, deletion, move from/to, formatting) becomes colour plus a `text-decoration` (`underline`, `underline double`, `line-through`, `line-through double`) or a highlight (the colour mixed 78% with white behind near-black text, and a 2 px bottom rule in the full colour). Glyphs are written into the markup (`<span class="gl" aria-hidden="true">`), so a change never depends on colour alone where the style gives a glyph. Change bars are a left border in the margin (`ChangeBars`, `ChangeBarColor`). `CoverPage`, `Footer`, `TypeTags`, `Title` and `Language` do what their names say. Colours are checked to be `#RRGGBB`; anything else is replaced by black, so a style from an untrusted source cannot reach the sheet. **Presets.** Blackline (the default): blue double-underlined insertions, red struck deletions, green double-struck/double-underlined moves, bars, cover page, footer. SideBySide: colour-blind-safe blue (#0072B2) and orange (#E69F00) highlights with `+`, `−`, `→`, `←` glyphs; no bars, cover page or footer; the side-by-side form picks it when the settings still hold the default blackline. Word: red underlined insertions and struck deletions, green moves, a bar. Developer: green and red backgrounds with `+`/`-`, purple moves. **Contrast of the preset colours on white** (WCAG 2.x ratios; 4.5 is AA for body text, 3 for large text and graphics). Blackline: insertion 8.59, deletion 4.00, moves 5.14, formatting 4.00. Word: insertions/deletions/formatting 6.48, moves 2.87. SideBySide and Developer put near-black text (`#1a1a1a`) on the tint, 12.2–14.6, so the mark colours (SideBySide orange 2.25 and grey 2.85 against white) are only the rule under the highlight and the glyph, never the text. Where text itself takes the mark colour and falls below 4.5 (Blackline deletion and formatting, Word moves), the mark is also a decoration and hidden text names it, so the information is not carried by colour; the values are in the style presets, not the writer. ## 2. Inline layout The new document's tree in order. Each node is a `div.blk.k-<kind>` (a real `table`/`tr`/`td` for tables) with `data-new` (new node id), `data-old` (the linked old node ids, space separated) and `data-link` (the alignment link). Its own text is one `p.t` (`white-space: pre-wrap`), a list label is a `span.lbl` beside it, other kinds get a `div.kl` label (Footnote, Image, …). Children nest inside the block. Text inside `p.t`: | Markup | Meaning | |---|---| | bare text | unchanged | | `<ins data-change=N>` | inserted text of change N | | `<del data-change=N>` | deleted text of change N, in place | | `<span class="fmt" data-change=N title=…>` | text whose formatting changed (the title says how) | | `<span class="mv to|from" data-role data-change id=mvt-N|mvf-N>` | the new / old place of a moved block, with an `a.mvlink` to the other place | | `<span class="sep">¶</span>` inside `del` | a node boundary inside deleted text (a merge) | | `<span class="pbrk">¶</span>` inside `ins`/`del` | a paragraph break that was inserted/removed (a split/merge at a block boundary) | | `<span class="ctl" data-cp=hex>` | a control character or unpaired surrogate HTML cannot carry | | `<span class="sr">`, `<span class="gl">`, `<span class="tag">` | hidden screen-reader label, glyph, type tag: not document text | Deleted blocks and the old place of moved blocks appear as `div.blk.ghost` (with `data-old`) where they used to be: before the next kept block, and ahead of blocks inserted just before it, so a removed block reads before its replacement. Attribute changes (style, list label, hashes) are `div.attrs > span.ae[data-change data-key data-old data-new]` under the block, shown as a label and unquoted values (`List number: 1.5 → 1.2`, `(none)` for a missing side); the machine-readable key and values stay in the attributes. Hidden content (comments, properties under the Legal settings) is not drawn; it is in the ledger section. A change that belongs to a link whose other side the settings hide is a ledger entry, not an element. ## 3. Side-by-side layout `table.sbs`: header cells "Old: name" / "New: name", one row per block in the merged order. A kept block is on both sides; deleted blocks only left, inserted only right, a moved block twice (old place left with `mv from`, new place right with `mv to`). A split or merge is one row with all its blocks. The left cell holds only unchanged and deleted text, the right only unchanged and inserted text; a formatting change shows on both sides. Container blocks with no text and no change get no row. Indentation is the tree depth (`d1`…`d5`). ## 4. How a page reads back The tests parse every page they write and check, per comparison and per preset and layout: 1. the text of each `p.t` with `ins` removed (and `del` kept) equals the old node's compared text; with `del` removed it equals the new node's — for every visible node, including splits and merges (blocks joined by U+001E) and ghosts; 2. every change is exactly the elements its edits call for: whole-node insert/delete = one `ins`/`del` with the change's text; edits = `ins`/`del`/`fmt` elements whose concatenated text equals the edits' new/old/formatted text (side-by-side drops U+001E); a move = one `from` and one `to`; each attribute edit = one `ae` with its key and values; no element refers to an unknown change; 3. hidden nodes are absent; nothing in the page can run or fetch anything. "Compared text" is the text under the settings (`ProjectedGroup`): hidden normalizations (typographic quotes, dashes, whitespace under Legal) stay folded and are counted in the ledger section rather than drawn. HTML validity is checked separately with the W3C Nu validator. ## 5. Escaping All text from documents, names, hashes, ledger details and diagnostics goes through one escaper. Text: `&`, `<`, `>` are entities; characters HTML cannot carry (C0 controls other than tab and LF, DEL and C1 controls, unpaired surrogates, noncharacters) become `span.ctl` with the code point in `data-cp` and a visible control-picture glyph, so reading the page back gives the original characters. Attribute values: `&`, `"`, `<`, `>` are entities and the uncarriable characters become U+FFFD. `Language` must be letters, digits and hyphens or it falls back to `en`. ## 6. Entry points and the registration seam ```csharp Redline.Create("v1.txt", "v2.txt", "redline.html"); // form from the extension Redline.Create("v1.txt", "v2.txt", "side.html", RedlineForm.HtmlSideBySide); var cmp = Comparison.Of("v1.txt", "v2.txt", CompareSettings.Legal); // inputs sniffed by content cmp.SaveAs("redline.html"); // settings' style cmp.SaveAs("redline.json"); cmp.SaveAs("redline.html", RedlineStyle.Developer); byte[] bytes = cmp.ToBytes(RedlineForm.Html); // or cmp.Save(stream, form) ``` `RedlineForm` is a small abstract class (name, extensions, preferred style, `Write`). A form passed explicitly needs no registration. `DocumentSources` and `Redline` hold two process-wide lists, filled by explicit calls and never by discovery: - `DocumentSources.Register(IDocumentSource, priority = 0)` (or `Redline.Register(source, priority)`): sources consulted before the built-in plain-text one, highest priority first and the newest first among equals; same `Name` replaces. A specialist source (the scan source) registers above the source that would also accept its input, so the order of `Register` calls does not matter. - `Redline.Register(RedlineForm)`: forms selected by extension; the latest registration of an extension wins; same `Name` replaces. Both are thread-safe and idempotent. An adapter package exposes one `Register()` that makes those calls. What a user writes: ```csharp PdfRedline.Register(); // once at startup; each adapter package has one Redline.Create("a.pdf", "b.pdf", "out.pdf"); ``` Without that call the failure names the package: `No output form is registered for '.pdf'. Reference RecursiveAI.OneDiff.Pdf and call its Register method once at startup.`, and an unreadable PDF input says it looks like a PDF and names the same package. Code that wants no global state uses `CompareDocument.From(input, source)`, `Comparison.Of(oldDocument, newDocument)` and a form's `Write` directly. A module initializer was rejected: it would run only after the adapter assembly is touched, which is exactly what `Redline.Create("a.pdf", …)` does not do, and library module initializers are discouraged for trimming and AOT. When no style is given: the form's `PreferredStyle` (side-by-side: `SideBySide`) if the settings' style is still the default blackline, otherwise the settings' style. ## 7. Command line `onediff compare <old> <new> <output> [--form html|html-side-by-side|json] [--settings legal|strict|text] [--style blackline|sidebyside|word|developer] [--title T] [--language tag] [--timestamp ISO-8601] [--no-cover] [--no-footer] [--no-bars] [--type-tags]` over `Redline.Create`. ## 8. Samples `docs/samples/<preset>-inline.html` and `<preset>-side-by-side.html` for the four presets, written from a short pair of documents made for the purpose (a replaced number, a deleted clause, an inserted clause, a moved clause that renumbers, a formatting change, a typographic difference in the ledger). A test compares them with what the writers produce now; `ONEDIFF_UPDATE_SAMPLES=1` rewrites them.  # PDF comparison How `RecursiveAI.OneDiff.Pdf` reads a digital PDF into the compare model, and how it marks a comparison on the original pages. Everything stated here is checked by the package's tests. ## 1. Reading a PDF `PdfDocumentSource` recognizes a PDF by its `%PDF-` header within the first 1024 bytes (ISO 32000-2 7.5.2), never by file name. It reads the text with OnePDF's extraction (glyphs, words, lines, blocks in reading order, header and footer roles) and builds the flowing compare model. ### Structure | Source content | Node | |---|---| | Body block | `Paragraph`; a block that starts with a bullet, a number or a letter marker (`•`, `-`, `1.`, `(a)`, `iv)`, or a dotted-decimal clause number such as `1.1` or `10.2.3.`) is a `ListItem` with the marker as the `list.label` attribute and the rest as its text | | A marker that the text analysis made a block of its own, beside a block on the same baseline | the same `ListItem` | | Block recognized as a table | `Table` → `TableRow` (a line) → `TableCell` (the words between two cell starts) → `Paragraph` | | Running headers and footers | a second `Section` after the body: one `Header` and one `Footer` node for a page whose header or footer text differs from the previous page's, holding its paragraphs. Pages that repeat the text add nothing, so a document with more pages does not gain a header per page | Body paragraphs are joined where layout cut them: - **Across pages:** the last paragraph of a page and the first of the next are one paragraph when the first does not end a sentence (`. ! ? : ;` and their CJK, Arabic and Devanagari forms, looking through closing quotes and brackets) and the second starts with a letter that is not upper case. - **Across columns:** within a page, a block that starts higher up the page than the previous one ended continues it under the same rule. - **Split-off lines:** a block that starts closer under the previous line than any paragraph gap (less than 0.4 line heights) continues it under the same rule. The text analysis splits a justified first line off this way. - A list item is continued the same way; a table never is. - **Grids the text analysis did not flag as tables:** when it returns each cell of a grid as a block of its own, consecutive short blocks (at most three lines and eight words each, not list markers) that fall into rows of the same number of cells (two or more), each cell overlapping the one above it, for at least two rows, become a `Table` of `TableRow` and `TableCell` nodes. A block right under a cell of the row (closer than half a line height) and overlapping it is that cell's next line, because a narrow cell wraps ("1 200" and "EUR"). Any block or row that breaks the pattern ends the grid; a grid with fewer than two complete rows stays paragraphs. The lines of one cell are joined by a space, not a line break (a word broken by a hyphen keeps its break, so the hyphenation fold joins it). A grid that opens a page, under the table that closed the page before, goes on in that table when it has as many columns, and a row it repeats as a heading (the same cell texts as the table's first row) is not a row. A first line of a block that is a running header's text, or a line alone at the top of a page, is left out of the grid reading, because a header right above a repeated heading can be read as part of the heading's first cell. - A clause number that opens a wrapped line inside a block starts an item only after a line that ended a sentence or is three words or fewer, because a reference to a clause ("see clause\n2.2.") can fall there. Reading the clause number as the label gives a Word file's `ListItem` and a PDF's the same shape, which is what lets a Word/PDF pair align. A decimal that starts a line (`3.5 million`) is read as a label too; that is the cost of the rule. Not in the model: images, vector graphics, annotations, form fields and document properties. ### Formatting Each word carries the formatting of its first glyph: `run.bold` (font name contains bold, black, heavy, semibold or demi), `run.italic` (italic or oblique), `run.font` (the font name up to the first `-` or `,`), `run.size` (points, rounded to half a point) and `run.color` (`#RRGGBB`, left out for black). A run goes on over the space to a following word with the same formatting. ### Ledger folds | Fold | Category | Where | |---|---|---| | line break inside a paragraph → one space (nothing between two characters of a script written without spaces: CJK, fullwidth forms, Thai) | `LineBreaks` | body and furniture | | hyphen (`-`, U+2010, soft hyphen) at the end of a line, preceded by a letter, followed by a lower-case letter on the next line → nothing | `Hyphenation` | body and furniture | | number equal to the page's number, the page's label or the page count, in a header or footer → `#` | `PageNumbers` | furniture | | quotes, dashes, runs of spaces | as for every source | body and furniture | Each fold keeps the original characters, so settings that do not hide a category show its differences as ordinary changes. ### Anchors A node's anchor has one quadrilateral per line (for a list item, the line includes the marker). For any other block it is the hull of the block's own words, one box per run of neighbouring words: a run ends where the line changes, the reading direction turns, or a gap wider than two line heights opens, because whatever lies in such a gap (another table cell or column) is not the block's. Its text is covered by segments, one per word, one per punctuation mark and one per space between two words on a line (cut as the text core's word tokenizer cuts; a space at a line or page break, or across a gap wider than two line heights, has none), each with one quadrilateral: **a change found by a word diff therefore begins and ends on segment boundaries, and its anchor covers exactly the changed text**. Quadrilaterals use the coordinates OnePDF reports: points, **upright page space** (origin at the lower left of the page as displayed, y up, the page's `/Rotate` and crop box applied; ISO 32000-2 7.7.3.3), a zero-based page index, and the corners in the order **lower left, lower right, upper right, upper left** of the text. A token's quadrilateral comes from its glyphs when the glyphs spell the word (left-to-right, horizontal text), otherwise from sharing the word's box evenly along the writing direction. In right-to-left lines the glyph order is not stated precisely enough to rely on, so the order whose glyph texts spell the word is found by trying the plausible ones. **Overlapping glyphs.** Glyph boxes of different words, or of other characters of the same word, can overlap on the page (a line of underscores running under the next word, a line of stars with text printed over it). A token's box is cut back, midway between its own outermost glyph centre and the foreign glyph's centre, until no glyph that is not the token's own has its centre inside it; a glyph from the line above or below is cut off vertically. Where a foreign glyph lies between two of the token's own glyphs, no box can leave it out, and a highlight takes it with them: the page genuinely has two texts in one place. ### Errors and limits `Read` throws `InvalidDataException` for an input it cannot read; `PdfSourceErrors.FailureOf` returns why: `Malformed`, `Encrypted` (no password, or a wrong one; `PdfSourceOptions.Password` supplies it) or `TooLarge` (`MaxPages` default 10,000; `MaxCharacters` default 64 million; OnePDF's own limits). A truncated or damaged file that OnePDF can repair is read as repaired. Cancellation throws `OperationCanceledException`. ## 2. Original-page markup `PdfRedline` (a sealed class and a `RedlineForm`; preset `PdfRedline.OnPages`) writes real markup annotations (ISO 32000-2 12.5.6.10 to 12.5.6.14) onto the pages of the new PDF with `MarkNewDocument`, and optionally onto the old one with `MarkOldDocument`. The PDF's own bytes must be the input the comparison read (their SHA-256 is checked against the document's source hash). Page content is never touched. | Change | On the new document | On the old document | |---|---|---| | Insert | Highlight, insertion colour | – | | Replace, Split, Merge | Highlight, insertion colour; pop-up holds the old text | StrikeOut, deletion colour; pop-up holds the new text | | Delete | Caret where the text was, deletion colour; pop-up holds the old text | StrikeOut | | Move | Highlight, move-to colour; pop-up holds the old text | Highlight, move-from colour | | Format | Underline, formatting colour; the first line names the old and new formatting | Underline | | Attribute (list label) | Squiggly, formatting colour; the first line names the attribute and both values | Squiggly | - **Colours and glyphs** come from `PdfRedline.Style`, default `RedlineStyle.SideBySide` (blue insertions, orange deletions, `+` and `−`). - **Where a caret goes.** For a deletion inside a paragraph, at the end of the text before it (or the start of the paragraph's text when the deletion is first). For a deleted paragraph, after the last text before it that has a place in the new document, or at the start of the document's text when there is none. - **Contents.** The first line names the change and starts with the style's glyph (`+ Inserted`, `+ Replaced`, `− Deleted`, `← Moved`, …); the text after the first line feed is the other side's text exactly (`PdfRedline.SplitContents`). `/T` is `Author` (default `OneDiff`). - **Names.** `/NM` is `onediff-c{change id}-p{page}`, with `.{n}` for further annotations of the same change on the same page (a change of more than 2048 quadrilaterals). A change spanning pages gets one annotation per page. Every annotation maps to one change; every change has annotations or is listed in `PdfMarkupResult.Skipped`. The only skipped changes are those of containers with no text of their own (a section, a table, a row), whose content carries its own changes. - **Quadrilaterals** on one line that follow one another are merged, so a phrase is one band. - **Pop-ups** sit to the right of the marked area when the page has room, else to its left, closed. - **Rotation.** Anchors are in upright space; the writer maps them to default user space through the page's crop box and `/Rotate`. - **Determinism.** The same comparison, PDF and settings give the same bytes on every platform. `/M` is written only when `ModifiedDate` is set; nothing reads the clock. Identifiers come from the change ids and OnePDF's hash-derived `/ID`. - **Saving.** Incremental by default: the output is the original bytes followed by an update, so signatures over the original stay in place where their permissions allow markup (12.8.2.2). A file that was rebuilt on opening cannot take an update and is rewritten whole; `PdfMarkupResult.SavedIncrementally` says which happened. `Incremental = false` always rewrites. - **Encrypted input.** Needs `PdfRedline.Password`; the output keeps the input's encryption. - **In the browser (WebAssembly).** The .NET browser runtime has no AES, so encrypted PDFs cannot be read or marked there; reading reports `Encrypted`. Everything else works in the browser. ### As an output form `PdfRedline.Register()` registers `PdfDocumentSource` and `PdfRedline.Auto` (name `pdf-auto`, extension `.pdf`) with the `Redline` statics, once and idempotently. **`Auto` follows the new document, and only the comparison** [Agent default]: when the new document's `SourceFormat` is `application/pdf` (or a scanned image, `image/png`, `image/jpeg`, `image/tiff`), it writes `PdfRedline.OnPages` (the new document with the marks, with the `SideBySide` look unless the caller or the settings chose one); for every other new document (text, Word) it writes the rendered redline of section 4. So `Redline.Create("a.pdf", "b.pdf", "out.pdf")` marks the new PDF's pages, `Redline.Create("a.txt", "b.txt", "out.pdf")` renders a blackline, and a pair with a PDF as the new document and a text file as the old one is marked on the PDF. Both forms stay selectable by name: `cmp.SaveAs("out.pdf", PdfRedline.Rendered)` renders a PDF pair, and `PdfRedline.OnPages` on a comparison whose new document is not a PDF throws. No registration order, no process state and no earlier call decides which one you get. The marking form needs the new document's bytes. They are kept on the document itself (`CompareDocument.Origin`, set by `PdfDocumentSource` and the scan source), in memory only: not hashed, not compared and never written to JSON. A comparison whose new document has no origin (one read back from JSON, or one built by hand) cannot be marked through `SaveAs`; `Auto` and `OnPages` throw `InvalidOperationException` saying so and naming the two ways out: `PdfRedline.OnPages.MarkNewDocument(comparison, bytes, stream)` with the PDF's bytes, or `PdfRedline.Rendered`. The style passed to `SaveAs` replaces `Style`. **Pages without text, and damaged files.** A PDF whose pages carry no text at all is a scan (or a drawing), and reading it as text would compare an empty document. `PdfDocumentSource` refuses it with an `InvalidDataException` whose `PdfSourceErrors.FailureOf` is `NoTextLayer` and whose message names `RecursiveAI.OneDiff.Ocr` and `OcrRedline.Register()`; the refusal does not depend on the settings. A PDF whose pages have no content stream at all (no text, and no image to read by OCR either) is not a scan but a truncated or damaged file: it is refused as `Malformed` with a message saying so, with or without the OCR package registered. A file from which no page could be read is `Malformed`. A file that opens but from which OnePDF could not recover some content comes with a warning on the document, which the comparison reports as the diagnostic `source.old.pdf.unrecovered` or `source.new.pdf.unrecovered` (and `CompareSettings.Strict` throws on it); repairs alone are an informational `pdf.repaired` diagnostic. ### Whole words on the page A word diff can end inside a word (a character-level refinement of a number or an identifier), and the text core reads some runs as one word (`10.8`, `a_b`). The page marks **whole tokens of the text core's word tokenizer**: a changed range that cuts a token is rounded out to the token. The other side's text in the pop-up is exact. A change that is only white space across a line break has no box of its own and is anchored on its block; the check has no text to compare for it. ## 3. Reading the marks back The tests re-extract every marked PDF with OnePDF and check, for every pair: each annotation has a change; each change has its annotations or a skip reason; the text under the quadrilaterals of the annotations of a change equals the change's new text (spaces, line-end hyphens and typographic variants aside, and with folded page numbers read as the digits the page shows); each pop-up's text after the first line equals the change's old text. The text under a change that cuts a token is compared with the change's text rounded out to the token, taken from the text as the page shows it. Where glyph boxes of different words overlap on the page and a highlight cannot avoid taking the other word (see Overlapping glyphs), the text under it may also be given with the overlapping words left out; the check accepts that, and only that, as equal. The gate set is run by `PdfGateRunTests` (environment `ONEDIFF_PDF_GATE_DIR`, `_FROM`, `_COUNT`, `_JOBS`, `_OUT`) over every pair, with the Legal and the Strict settings: verification passed with no fallback and a clean re-check; the marks re-extracted as above on the new document, and the old document's marks likewise; a second run giving byte-identical comparison JSON and marked PDFs. ## 4. Rendered redline `PdfRedline.Rendered` (inline) and `PdfRedline.RenderedSideBySide` are `RenderedPdfRedline` forms: a new PDF laid out from the comparison alone, for any kind of input, plain text included. The package builds the same page as the HTML redline (`docs/output-html.md`: cover page, key, marks, ledger, footer with the input names and hashes) with a style sheet for paper, and renders it with OneRender's `HtmlPdf.ToPdf`. `Render(comparison, stream, style)` is `Write` plus a result with the page count and the renderer's notes. - **Layout.** A4 (`Paper = PdfPaper.Letter` for Letter: `new RenderedPdfRedline { Paper = PdfPaper.Letter }`), margins 18 mm and 16 mm, the running page number in the bottom margin. Side by side is landscape: one grid row per block, old on the left and new on the right, and a row is kept whole on a page when it fits on one (see the limits below). The cover page, change bars and footer follow the style (`CoverPage`, `ChangeBars`, `Footer`); the timestamp only when the style carries one. Each block takes its direction from its first strong character (`unicode-bidi: plaintext`). - **Look.** The comparison settings' style unless the caller passes one; `Blackline` for the inline layout and `SideBySide` for the side-by-side layout when the settings left the style at the default. Marks are drawn with the style's colours, decorations (underline, double underline, strike-through, highlight) and glyphs. A control character the page shows as a control picture takes the colour of the change it is in. - **Static.** The page uses no script, no network and no font outside the ones the renderer carries (Liberation Sans). The text-for-screen-readers labels of the HTML page ("inserted:", "deleted:") are not written, since the colour, decoration and glyph of the style carry the meaning and the key explains them. The page's closed ledger sections are written open. - **Scripts.** The renderer draws a script only when a font that covers it is referenced by the application: `RecursiveAI.OneRender.Fonts.Cjk`, `.Fonts.Noto.Arabic`, `.Fonts.Noto.Devanagari` and the other `.Fonts.Noto.<Script>` packages, `.Fonts.Emoji`. Each package registers itself when the application references it; nothing in `RecursiveAI.OneDiff.Pdf` changes. Without a font the characters are not drawn but stay in the text layer, so extraction and search still find them (the renderer's notes do not say so; see the limits). - **Tagged.** The output is a tagged PDF with the document language (`RedlineStyle.Language`) and title (`RedlineStyle.Title`, or "Comparison of *old* and *new*") set; headings, paragraphs, the counts and ledger tables and the lists carry the renderer's structure types. - **Deterministic.** The same comparison and style give the same bytes: no creation date is written and the clock is never read. - **Limits** (renderer 1.3.3; `OneRenderGapTests` pins what is pinned with a minimal case): - a table row whose cells differ in height can lose a line at a later page break (a 40-line cell beside a 100-line cell lost its 97th line). The side-by-side page therefore keeps each row whole where it fits on a page and does not repeat the column headings on each page; a row taller than a page can still lose a line at its cut; - a page list holds a few million glyphs: a redline of that size (the largest text pair of the test corpus, 29 MB) fails with an `InvalidOperationException` that names the forms without the limit (HTML, JSON, markup on the original pages); - a colour emoji needs `RecursiveAI.OneRender.Fonts.Emoji` referenced, like every other script. ### Fonts for other scripts `RecursiveAI.OneDiff.Pdf` and the `onediff` tool reference no script font package, so only Latin (Liberation Sans, which the renderer carries) is drawn; the characters of other scripts stay in the text layer but are not drawn, and the renderer's notes say so (`RenderedPdfResult.Notes`, one line per missing glyph run). To draw them, reference the font packages the documents need in the application, for example `RecursiveAI.OneRender.Fonts.Cjk`, `RecursiveAI.OneRender.Fonts.Noto.Arabic`, `RecursiveAI.OneRender.Fonts.Noto.Devanagari`; each registers itself, and nothing else changes. The tests reference those three. ### Bidirectional controls The PDF source keeps the extraction as it comes, including U+200C/U+200D and the bidirectional control characters, and does not set `PdfTextOptions.RemoveBidiControls`: the extractor reads right-to-left runs in logical order, and the comparisons made with the controls kept verify and show no changes that are only controls on the 1,110 test pairs. ### Reading the rendered pages back `RenderedGate` (in the tests) re-extracts a rendered PDF with OnePDF and checks it against the comparison. For an inline page it puts every glyph in a class by its fill colour (inserted, deleted, moved from, moved to, formatting, plain; a style whose colours differ is needed, and the gate uses the blackline with its own colour for moves from). Without the classes that are not in the new document it must hold the new text, and without those not in the old document the old text, in order, block by block; each edit's inserted text must be in the inserted class and its deleted text in the deleted class. For a side-by-side page the left half must hold the old blocks and the right half the new blocks. Text is compared with whitespace and joiners left out, as the page shows it (control characters as their control pictures, the folds of categories that are not hidden undone). Lines are read in the order a reader takes them: bands cut at the vertical gaps, columns cut at horizontal gaps that a line from the other side does not cross (the cells of a table row one after another), and a side-by-side page column by column. A document with right-to-left text anywhere is still checked as a bag of characters, because the neutral characters between mixed-direction runs depend on the bidirectional algorithm. What the extraction cannot see: underlines, strike-throughs and highlights are drawn as shapes, so a style whose marks differ only by decoration (the Word preset's insertions and deletions are both red) is checked for text only; the page's colour is the only styling it reads. The text corpus and the PDF pairs are run by `RenderedGateRunTests` (`ONEDIFF_RENDER_TEXT_CORPUS`, `ONEDIFF_PDF_GATE_DIR`, `ONEDIFF_RENDER_FROM`, `_COUNT`, `_EVERY`, `_JOBS`, `_OUT`); `tools/run-render-gate.sh` runs them in queued chunks.  # Scans and images How `RecursiveAI.OneDiff.Ocr` reads a scanned PDF or an image into the compare model, what the possible-OCR-error flag means, and how the scan gold set is built and scored. ## 1. What is read as a scan `OcrDocumentSource` reads PNG, JPEG and TIFF images (recognized by their signatures, never by file name) and PDFs whose pages carry no text of their own. - A PDF is a scan when none of the pages examined has at least `MinTextLayerCharacters` (default 20) non-space characters in its text layer. All pages are examined up to 8; beyond that, 8 pages spread evenly. A PDF none of whose pages has a content stream is not a scan but a damaged file, and is left to the PDF source, which refuses it. When the pages render but the engine recognizes no word on any of them, the document carries the warning `ocr.nothingRecognized` (the diagnostic `source.<side>.ocr.nothingRecognized`), and strict settings throw on it. - A PDF with an invisible recognized-text layer therefore counts as a digital PDF and is read by the PDF source. - An unreadable or encrypted PDF is not a scan; the PDF source reports why. - Each image frame is one page. An image is wrapped in a one-page-per-frame PDF so that markings can be written on it. The marked file is checked against the wrapper's hash (`PdfOrigin.Sha256`), the compare model's `SourceHash` is the image's own hash. `OcrRedline.Register()` registers the PDF source and the scan source, the scan source with a higher priority (`DocumentSources.Register(source, priority)`), so a scanned PDF goes to OCR whichever of `PdfRedline.Register()` and `OcrRedline.Register()` was called last. Without it, the PDF source refuses a PDF that has no text layer and says to reference this package. The `onediff` tool registers it. ## 2. Reading the text The recognizer returns pages of blocks, lines and words, each with a quadrilateral and a confidence. The model is built the way the PDF source builds it from a text layer, with these differences, which exist because two scans of one page are never segmented the same way: - **Block boundaries are soft.** A block that starts right under the open paragraph (a gap of at most 0.9 line heights, in the same column and type size) continues the paragraph, unless the paragraph's last line was a short line that ended a sentence. Where the recognizer cuts running text into blocks is its own choice and differs from scan to scan, so it is not a paragraph break. Inside a block a gap wider than 0.9 line heights (and, with four or more gaps, 1.6 times the median gap) ends a paragraph. - **Clause numbers start list items.** A line that starts with a multi-level number (`2.3`, `4.1.2`) followed by text is a list item with the number as its `list.label` attribute, whatever the spacing, so renumbering reads as a cascade and not as an edit. A clause number that follows a sentence end inside a line also starts one. - **Tables are off by default** (`ReadTables`). The recognizer finds the same table on two scans of a page only some of the time; a table on one side and running text on the other would be a difference of reading. The cell text still reads in order and every difference in it is shown. Set `ReadTables` to get table, row and cell nodes. - **Running furniture.** A line in the top or bottom tenth of a page whose letters resemble (Dice similarity ≥ 0.65 over letter pairs) a line in the same margin of another page is a header or footer. So is a line of digits and punctuation alone, and the words for "page" or "page of" in English, German, French, Spanish, Italian, Dutch, Swedish, Polish and Turkish. One node stands for each running text; the copy read with the highest confidence is the one kept. Page numbers fold to `#`, joined with the characters a scan puts between them (`Page 2/.3`, `Page 2 / 3`); the fold is a ledger entry (`PageNumbers`). - Line breaks and end-of-line hyphenation fold into the ledger as for a PDF (`LineBreaks`, `Hyphenation`). Nothing the recognizer read is corrected or hidden: a recognition error is a change like any other, with the flag below. ### Anchors Every token is anchored by `AnchorKind.OcrWords`: the quadrilaterals of the words (upright page space, points, origin at the lower left of the page as displayed, y up) and the word's confidence. A paragraph's anchor is one box per line. Offsets are UTF-16 offsets into the node's text. ## 3. The possible-OCR-error flag `Change.OcrConfidence` is the lowest confidence over the words the change's edits touch, resolved per edit range on each side (words next to the edit that merely sit in the same unit do not count). A whole inserted or deleted block has no edits; every word of its side counts. `Change.PossibleOcrError` is true when it is below `CompareSettings.OcrConfidenceThreshold`. - The flag never hides, drops or reorders a change. - Moves, formatting and attribute changes have no confidence. - JSON: optional `ocrConfidence` and `possibleOcrError` per change, `settings.ocrConfidenceThreshold` when not the default. Schema version stays 1. - **Default threshold 0.95**, chosen on the dev scan set only (see §5): a lower threshold has higher precision and lower recall. | threshold | precision | recall | |---|---|---| | 0.80 | .946 | .408 | | 0.90 | .939 | .484 | | 0.95 | .898 | .588 | | 0.97 | .877 | .676 | | 0.98 | .869 | .717 | | 0.99 | .851 | .835 | | 0.995 | .844 | .872 | Precision is the share of flagged changes that no real edit explains; recall is the share of such changes that are flagged. ## 4. Gold set method (scan-v1) `tools/OneDiff.GoldSets.Scan build` takes the PDFs of the digital gold set and scans each file of a pair independently: it renders at 200 or 300 dpi, blurs (sigma 0, 0, 0.7 or 1.2 at 300 dpi), skews by up to ±2°, adds salt-and-pepper noise (density 0, 0, 0.002, 0.006) and encodes each page as PNG or JPEG (quality 90, 75 or 55). All choices come from a seeded generator keyed by the file's name and the seed (20261104), so the set is reproducible. Languages: the Latin-script ones (en, de, fr, es), because the bundled recognizer reads Latin, Greek and Cyrillic. The labels are the digital set's own. A scan's nodes are matched to the gold blocks by similarity; positions are carried over by a minimal character alignment of the whole page's text against the gold text, so a gold block that the recognizer cut into several nodes, or several that it joined, still receive their changes. The scorer's noise rows count a **spurious change**: a text change that overlaps no gold unit, or a move of a block no gold link moves. ## 5. Rows and cost The rows are those of the digital gold sets with the same definitions, plus `noise.changes.share` (spurious changes over all changes), `noise.changes.per10kchars` and `ocr.flag.P/R/share`. The results are in `results/scan-v1-eval.md`. A scan costs the recognizer's time plus the comparison. The recognizer reads about 0.5 s per page on one core of the reference machine; `results/scan-v1-eval.md` has the budget measurement.  # Results The measured results of the 1.0 packages (`RecursiveAI.OneDiff`, `.Documents`, `.Pdf`, `.Ocr` and the `onediff` tool): the text core's exact gates, fuzzing and speed against the managed comparators; the PDF and scan engine's gates and gold-set scores; change semantics and combine; budgets; platforms. Losing rows and gaps are listed, not left out. Nothing is claimed beyond this page. Method: [measurement.md](https://recursiveai.org/onediff/docs/measurement/). Status words: **pass** meets the bar; **FAIL** a gate with failures (listed with its count); **floor** a score recorded as the floor later releases may only raise; **loses** a row where a comparator scores better; **information** measured, no bar. No commercial comparison product is measured or compared against. The adapter packages for Word, spreadsheets, presentations, rendering, OneNLP recognizers and the model ship later; their rows are not part of this page, except the DOCX comparator scores kept at the end as information. ## What was run | | x64 | arm64 | |---|---|---| | CPU | AMD Ryzen 9 7950X3D, 16 cores / 32 threads (avx, avx2, avx512f, avx512bw, bmi2, fma, popcnt, sse4.2) | Apple M2, 8 cores (arm64) | | OS | Linux (x86_64) | macOS (arm64) | | SDK / runtime | .NET SDK 10.0.403, runtime 10.0.14 | .NET SDK 10.0.403, runtime 10.0.14 | | Timed rows | alone on the machine's benchmark queue; load average before the runs 1.6–6.5 (1 min), falling to 1.8–2.5 while they ran (other users' idle processes) | the Mac carries a steady background load of about 2–4 and the benchmark cannot raise its priority: these rows are less quiet than the x64 rows | Code under test: the release candidate plus test-and-tooling changes only (no library code changed). Sister libraries: the rows were measured on the OneFont 1.4.0 and OnePDF 1.18.0 release candidates; the release builds against the published OneFont 1.4.0, OnePDF 1.18.0, OneImage 1.10.5, OneRender 1.3.3 and oneocr 1.0.0, and its test suites passed on those on both machines. Comparators and reference tools run as released binaries and packages: DiffPlex 1.9.0, DiffMatchPatch 5.0.0, git 2.51.2, GNU diffutils 3.12, GNU patch 2.8, the `diff-match-patch` Python package 20241021, W3C Nu Html Checker 26.11.4, BenchmarkDotNet 0.15.8. Data sets (all evaluation splits, never used for tuning): text corpus manifest sha256 `087bd93a…` (9,483 code and prose pairs, 2,599 merge triples; pinned fetch in `tools/corpora-pins.json`); `synth-v1` `e11b5614…` (1,050 pairs, 7 languages); `pdf-v1` `eb10c6b3…` (1,110 pairs rendered to PDF, 60 of them real); `scan-v1` `6fe36af3…` (600 scanned pairs); `combine-v1` `5c55ebb5…` (630 cases; regenerating it from the seed gives the same hash). ## Exact gates (text core) Oracles run as black-box processes. Zero failures in every gate. | Gate | Result | Reference | Verdict | |---|---|---|---| | Myers edit length is the minimum (`diff --minimal` length) | 9,483 / 9,483 | GNU diffutils 3.12 | pass | | Every algorithm's script transforms old into new (Myers, patience, histogram; line, word, character tokens) | 9,483 / 9,483 | — | pass | | Unified patch round trip (make, parse, apply) | 9,483 pairs, 0 failures | — | pass | | dmp patch round trip (make, text, parse, apply) | 9,483 pairs, 0 failures | — | pass | | Our unified output accepted by GNU patch | 9,481 accepted, 2 skipped, 0 failures | GNU patch 2.8 | pass | | Our unified output accepted by `git apply` | 9,481 accepted, 2 skipped, 0 failures | git 2.51.2 | pass | | dmp patch text interop with the Python package, both directions | 9,454 agree, 29 skipped (inputs too large for that package), 0 failures | diff-match-patch 20241021 | pass | | Three-way merge equals `git merge-file -p`, byte for byte, conflict style `merge` | 2,599 / 2,599 | git 2.51.2 | pass | | Same, conflict style `diff3` | 2,599 / 2,599 | git 2.51.2 | pass | | Same, conflict style `zdiff3` | 2,599 / 2,599 | git 2.51.2 | pass | The skipped cases are listed in the gate output; none is a failure. Cases skipped by the oracle (timeouts, over-size) are counted as skips, never passes. ## Fuzzing Seeded mutation runners (`fuzz/OneDiff.Fuzz`, no coverage feedback): inputs are mutations of patches the library writes plus hostile cases; for the change-list reader, mutations of valid comparisons; for the PDF source, mutations of valid PDFs (flips, cuts, splices, repeated chunks, extreme numbers). A crash is an exception other than the documented typed failure, a broken invariant, or an input slower than 3 s. Seed 1, release build, one process per target, all seven run side by side on the x64 machine (8 cores reserved; load average 11–16). | Target | Duration | Executions | Crashes | |---|---|---|---| | unified-parse | 600 s | 368,289,410 | 0 | | unified-apply | 600 s | 320,255,298 | 0 | | dmp-parse | 600 s | 400,521,790 | 0 | | dmp-apply | 600 s | 358,334,639 | 0 | | unified-roundtrip | 600 s | 82,887,489 | 0 | | dmp-roundtrip | 600 s | 10,275,281 | 0 | | comparison-json (change-list reader) | 600 s | 43,825,170 | 0 (slowest input 23.4 ms) | | pdf-source (new for this release: damaged and image-only PDFs are refused) | 60 s smoke | 23,310 (13,362 read, 9,948 refused with the typed failure) | 0 | ## Document exact gates Every pair is compared with the Legal and the Strict settings. | Gate | Result | Verdict | |---|---|---| | `synth-v1` eval: Verify passes, fallback level 0 | 1,050 / 1,050 pairs, fallback 0 | pass | | `pdf-v1`: Verify, fallback 0, `Check()` clean | 2,220 comparisons, 0 failures | pass | | `pdf-v1`: marks on the new document re-extracted with OnePDF equal the change list | 2,220 comparisons, 0 failures | pass | | `pdf-v1`: marks on the old document, re-extracted likewise | 2,220 comparisons, 0 failures | pass | | `pdf-v1`: a second run gives byte-identical comparison JSON and marked PDFs | 2,220 comparisons, 0 failures | pass | | `scan-v1`: Verify, fallback 0 | 600 pairs × Legal/Strict, 0 failures | pass | | `scan-v1`: flags and confidences round-trip through JSON | 32,646 changes (17,777 flagged), 0 failures | pass | | `scan-v1`: marks on the new and old pages re-read | 54,334 marks, 0 failures | pass | | `scan-v1`: determinism (two runs, same bytes) | 100 pairs, 0 failures | pass | | Rendered redline PDF (OneRender 1.3.3) read back equals the change list, inline and side by side each: text corpus | 638 pairs, 0 failures | pass | | The same, text-corpus pairs over 300 kB | 24 pairs, 0 failures | pass | | The same, `pdf-v1` | 1,110 pairs, 0 failures (the first run of the release gate failed 85 pairs, inline layout only; fixed, see below) | pass | | HTML redlines parsed back: every mark carries its change, text equals the change list | 96,000 generated pages (12,000 comparisons × 8 styles and layouts) and 1,324 corpus pages, 0 mismatches | pass | | HTML pages against the W3C Nu Html Checker 26.11.4 | 370 pages, 0 errors, 0 warnings | pass | | Seeded invertibility sweep (random edit sequences) | 60,000 cases, 0 failures | pass | | Seeded alignment sweep (labelled and table pairs) | 60,000 cases, 0 failures | pass | | Seeded grouping sweep (every preset) | 60,000 cases, 0 failures | pass | | Seeded semantics sweep (random contracts: complete, groups partition the changes) | 60,000 cases, 0 failures | pass | | Combine sweep, 1 to 4 reviewers | 60,000 cases, 0 failures | pass | | Chain sweep, 3 to 5 steps | 20,000 cases, 0 failures | pass | **The rendered `pdf-v1` failure, fixed.** A table cell whose text wraps over two lines in the source PDF (for example "6" and "months") kept the line break in its node text; the inline page drew the cell as a stack of short lines, and the re-extraction read the neighbouring cell's text between the cell's lines. The grid reading now joins the lines of a cell with a space (a word broken by a hyphen keeps its break so the hyphenation fold still joins it). The smallest failing pair is a test (`RenderedReadbackTests`). Re-run on the fixed code: `pdf-v1` Verify, fallback 0, `Check`, marks re-read and determinism on 1,110 pairs × Legal/Strict, the rendered read-back inline and side by side on 1,110 pairs, and the text corpus (638 pairs and 24 over 300 kB): 0 failures. The PDF digest slice (20 pairs) is identical on x64 and arm64. **After the cell-text pairing rule** (a paired table cell with one unmatched text block on each side pairs them), the rows it can touch were scored again on the final code: `synth-v1` eval (above: edited, unrelated and changes per gold unit improved, every other row identical), change semantics (every row identical), `combine-v1` (every row identical, digest unchanged), `scan-v1` (every row identical, 600 pairs), the `scan-v1` exact gates on 100 pairs (every 6th; 5,322 changes, 8,921 marks, 20 determinism re-checks, 0 failures), and the alignment, invertibility and grouping sweeps at 60,000 cases each (seeds 1–60,000 in two chunks), 0 failures. ### Same bytes on both machines A fixed slice of `pdf-v1` (20 pairs, every 56th) and of `scan-v1` (20 pairs, every 30th), plus generated and corpus inputs. Each file below has the same SHA-256 on x64 and arm64. | Digest | SHA-256 | Equal | |---|---|---| | `pdf-v1` slice: inputs, comparison JSON, marked PDFs (new and old) | `e8c27c0e…` | yes | | `pdf-v1` slice: rendered redline PDFs, inline and side by side | `f877ec66…` | yes | | Rendered text corpus (every 37th pair), PDFs | `21f32088…` | yes | | Generated inputs, comparison JSON and marked PDFs (10 seeds) | `b2a02d49…` | yes | | `scan-v1` slice: JSON and marked PDFs (OCR source) | `1a421327…` | yes | | HTML pages (random documents in every style and layout, corpus pairs) | `2f32fe58…` | yes | | `combine-v1` first 20 cases: combination JSON and report pages, two runs each | `2c58a823…` | yes | | `synth-v1` slice: comparison JSON of 20 evaluation pairs (4 documents × 5 variants: en, de, zh, ar) | `ab579d71…` | yes (also equal to the digest before the cell-text pairing rule) | ## Gold sets Paired bootstrap over documents, 10,000 resamples, seed 20261104, 95% percentile intervals. Every row is scored once on the evaluation split. All rows equal or beat their floors except one, marked. ### Alignment, boundaries, fragmentation (`synth-v1` eval) Gold documents as input; 210 documents, 1,050 pairs. | Row | Score [95% CI] | Floor | Verdict | |---|---|---|---| | align F1, same | 0.9999 [0.9998, 1.0000] | 0.9999 | floor held | | align F1, edited | 0.9881 [0.9834, 0.9924] | 0.979 | above floor (was 0.9793) | | align F1, moved | 0.9880 [0.9749, 0.9979] | 0.988 | floor held | | align F1, split | 0.9960 [0.9862, 1.0000] | 0.996 | floor held | | align F1, merged | 1.0000 [1.0000, 1.0000] | 1.000 | floor held | | align F1, unrelated | 0.9552 [0.9360, 0.9717] | 0.928 | above floor (was 0.9278) | | boundary F1 | 0.9144 [0.9036, 0.9246] | 0.914 | floor held | | boundary precision / recall | 0.9135 / 0.9152 | .912 / .911 | floor held | | fragmentation: changes per changed sentence | 1.0047 [1.0028, 1.0070] | 1.005 | floor held | | fragmentation: changes per gold unit | 1.1553 [1.1286, 1.1847] | 1.171 | better than floor (a lower count is better; was 1.1712) | | display regions per changed sentence / per unit | 1.0047 / 1.0164 | — | information | | missed changed sentences | 0.0018 [0.0003, 0.0038] | 0.0018 | floor held | | Verify passes / fallback rate | 1.0 / 0 | 1.0 / 0 | pass | Fragmentation counts changes, not display regions, and is read beside boundary precision (definition in [measurement.md](https://recursiveai.org/onediff/docs/measurement/#scoring)). These are gold-set rows with floors, not comparator rows. ### Change semantics (`synth-v1` eval) | Row | Score [95% CI] | Floor | Verdict | |---|---|---|---| | type accuracy, all typed units | .9986 [.9971, .9997] | .9986 | floor held | | resolved-value accuracy, all typed units | .9983 [.9966, .9997] | .9983 | floor held | | amount type / value | 1.0 / .9965 ([.9887, 1]) | same | floor held | | date, duration, percentage, defined term, number (type and value) | 1.0 | 1.0 | floor held | | reference | .9869 [.9727, .9970] | .9869 | floor held | | cascade group precision / recall | .9778 [.9600, .9933] / .9809 [.9642, .9940] | .978 / .981 | floor held | | rename group precision / recall | 1.0 / 1.0 | 1.0 / 1.0 | floor held | | substantive filter precision / recall | .9986 [.9971, .9997] / .9989 [.9971, 1.0] | .9986 / .9989 | floor held | Known limit: the remaining cascade misses are Japanese reference edits merged with a neighbouring adverb change into one unit. ### Combine and chains (`combine-v1`) 630 cases from the 210 evaluation documents with 2, 3 and 4 reviewers. | Row | Score [95% CI] | Floor | Verdict | |---|---|---|---| | invertible per reviewer, check passes, fallback 0 | 630 / 630 | 630 / 630 | pass | | attribution accuracy | .9937 [.9910, .9962] | .9937 | floor held | | shared-edit merge precision / recall | .9787 [.9610, .9927] / .9956 [.9886, 1.0000] | .9787 / .9956 | floor held | | conflict precision / recall | .9648 [.9453, .9819] / .9843 [.9723, .9953] | .9648 / .9843 | floor held | ### Reading a PDF (`pdf-v1`, every fifth pair) The PDF source run on 210 synthetic pairs (222 compared, 0 failures), its nodes mapped onto the gold blocks (20,689 of 22,191 content nodes matched a gold block; the rest are page furniture and extraction artefacts, which count as unrelated). | Row | Score [95% CI] | Floor | Verdict | |---|---|---|---| | align F1, edited | .9361 [.9171, .9542] | .909 | above floor | | align F1, moved | .8429 [.7460, .9241] | .843 | equal at three decimals (the interval's centre is unchanged; moved precision .756, recall .952) | | align F1, split | 1.0000 | .781 | above floor | | align F1, merged | .9706 [.9206, 1.0000] | .846 | above floor | | align F1, unrelated | .5386 [.4756, .5966] | .263 | above floor (low because page furniture has no gold block) | | boundary F1 | .8297 [.7994, .8580] | .792 | above floor | | fragmentation: changes per changed sentence | 1.0057 [1.0012, 1.0109] | 1.006 | floor held | | fragmentation: changes per gold unit | 1.1735 [1.1325, 1.2185] | 1.174 | floor set (re-based from 1.135; a lower count is better): the earlier value counted wrong matches of short equal cells across table rows. Since grids are read as tables, an inserted or deleted row costs one change per cell, while boundary F1 and alignment rose. One change per row is planned | | missed changed sentences | .0423 [.0269, .0598] | .108 | above floor | | Verify passes / fallback | 1.0 / 0 | 1.0 / 0 | pass | ### Scanned pairs (`scan-v1`) 120 evaluation documents (30 per language: en, de, fr, es), 600 pairs. "Digital" is the same pairs read from the digital PDFs by the PDF source, for comparison; the digital column is information. | Row | Scan [95% CI] | Floor | Digital | |---|---|---|---| | align F1 same | .7041 [.6949, .7131] | .704 | .9569 | | align F1 edited | .4997 [.4700, .5281] | .500 | .9289 | | align F1 moved | .3205 [.2582, .3814] | .321 | .8443 | | align F1 split / merged | .0552 / .1085 | .055 / — | .9867 / .9905 | | align F1 unrelated | .0524 [.0430, .0625] | .052 | .4658 | | boundary F1 | .2367 [.2155, .2589] | .237 | .8508 [.8300, .8705] | | boundary precision / recall | .1400 / .7671 | .140 / .767 | .7921 / .9190 | | frag.sentence / frag.unit | 1.0464 / 1.1115 | 1.046 / 1.112 | 1.0067 / 1.1802 | | missed changed sentences | .0945 [.0762, .1139] | .095 | .0100 | | noise.changes.share | .8649 [.8503, .8782] | .865 | .4314 | | ocr.flag.P (flagged changes no real edit explains) | .9320 [.9224, .9406] | .932 | — | | ocr.flag.R (such changes that are flagged) | .6197 [.5889, .6492] | .620 | — | | Verify passes / fallback | 1.0 / 0 | 1.0 / 0 | 1.0 / 0 | The scan boundary and noise rows are low because recognition errors make most changes "noise" (86%), a limit of the recognition engine; the flags mark such changes (precision .93, recall .62 at threshold .95). The digital column is lower than at the previous measurement (boundary F1 .851 against .879, edited .929 against .944, unrelated .466 against .438; intervals overlap); it is the PDF source reading the same pairs after later changes to clause numbers and tables, and it is not a floor row. ## Speed: text core against the managed comparators Comparators: DiffPlex 1.9.0 and DiffMatchPatch 5.0.0 (the diff-match-patch authors' C# package), restored in the benchmark project only. Ratio = ours ÷ comparator, so lower is better. Bar: geometric mean per comparator and area at most 1.0, no row above 1.25. **Settings.** Single-threaded, workstation non-concurrent GC; DiffMatchPatch `Diff_Timeout = 0`; no cleanup in the diff rows; DiffPlex `ignoreWhitespace: false` with an explicit word separator set; DiffMatchPatch's line mode maps lines to characters inside the timed work (its helper is not public); cleanup rows time the copy of the input diff on both sides. BenchmarkDotNet 0.15.8, one launch per row with its default warm-up and iteration counts, `Release`. Inputs are corpus pairs of about 2 KB (Small), 40 KB (Medium) and 300–400 KB (Large; lines and words only), named in the raw tables. Each area ran as its own group of under 20 minutes. Raw BenchmarkDotNet output, the machine record and the input and edit-distance listing are kept with the run; peak working set is measured once per row in a cold child process. | Area | Comparator | x64 geomean | x64 worst row | arm64 geomean | arm64 worst row | Verdict | |---|---|---|---|---|---|---| | lines (3 algorithms × 3 sizes) | DiffPlex lines | 0.19 | 0.32 | 0.19 | 0.34 | pass | | lines | diff-match-patch line mode | 0.52 | 0.72 | 0.54 | 0.78 | pass | | words (3 × 3) | DiffPlex words | 0.29 | 0.81 | 0.35 | 0.93 | pass | | chars (3 × 2) | DiffPlex characters | 0.01 | 0.08 | 0.01 | 0.08 | pass | | chars | diff-match-patch `diff_main` | 0.22 | 0.56 | 0.24 | 0.67 | pass | | cleanup, semantic (2 sizes) | diff-match-patch | 0.32 | 0.48 | 0.33 | 0.48 | pass | | cleanup, efficiency (2 sizes) | diff-match-patch | 0.51 | 0.89 | 0.53 | 0.93 | pass | | patch apply (2 sizes) | diff-match-patch | 0.61 | 0.65 | 0.61 | 0.64 | pass | | patch make (2 sizes) | diff-match-patch | 0.85 | **1.13** | 0.83 | **1.02** | pass (geomean under 1.0; worst row under 1.25) | | patch toText | diff-match-patch | 0.21 | 0.32 | 0.18 | 0.26 | pass | | patch fromText | diff-match-patch | 0.09 | 0.11 | 0.07 | 0.08 | pass | Three-way merge has no managed comparator and is covered by the exact gate above, not a speed row. - **Words are not equal work.** DiffPlex's word differ cuts words at the separators it is given, so it does not see our token stream: it reports fewer edits on the same input (Small 48 against our 57–61, Medium 60 against 69, Large 840 against 1,114). The ratios compare the time to produce each library's own script. Lines and characters agree on edit distance between all implementations except where a heuristic algorithm (patience, histogram) finds a longer, equally valid script than Myers. - **Not covered.** No word rows against diff-match-patch (it has no word mode), no large-size character rows, no large-size cleanup or patch rows, no multi-threaded rows (the text core is single-threaded). Native tools (git, GNU diff) are not compared. ### Allocations (bytes per call, x64; arm64 agrees within 1%) | Row | Ours | Comparator | Ratio | |---|---|---|---| | lines, Medium, Myers | 105,960 | DiffPlex 317,592 / diff-match-patch 342,648 | 0.33 / 0.31 | | lines, Large, Myers | 960,825 | DiffPlex 2,132,880 / diff-match-patch 2,880,414 | 0.45 / 0.33 | | words, Large, Myers | 3,031,091 | DiffPlex 10,878,894 | 0.28 | | chars, Medium, Myers | 1,960 | DiffPlex 3,637,006 / diff-match-patch 229,568 | 0.00 / 0.01 | | semantic cleanup, Small | 1,216 | diff-match-patch 21,512 | 0.06 | | efficiency cleanup, Medium | 0 | diff-match-patch 328 | 0.00 | | patch make, Medium | 253,104 | diff-match-patch 392,096 | 0.65 | | patch apply, Medium | 325,328 | diff-match-patch 487,464 | 0.67 | Every measured row allocates less than its comparator (allocation ratio at most 0.67). ### Memory Peak working set above the inputs, x64, cold child process: | Row | Ours | DiffPlex | diff-match-patch | |---|---|---|---| | lines, Large (Myers) | 5.7 MB | 5.3 MB | 4.0 MB | | words, Large (Myers) | 14.7 MB | 12.7 MB | — | | chars, Medium (Myers) | 2.3 MB | 7.5 MB | 2.8 MB | At the two largest rows where we are higher the process baseline dominates (within about 8–16% of DiffPlex's); no bar is set on peak memory for the text core. **Linear growth** (lines, Myers, bytes allocated per diff on inputs of 2 KB, 40 KB and 317 KB): 9,144, 230,184 and 1,514,568, that is 4.6, 5.8 and 4.8 bytes per input byte. The time grows the same way (1.7 µs, 68 µs and 792 µs). ## Budgets A 100-page pair with about 300 changes, 4 cores (`DOTNET_PROCESSOR_COUNT=4`), one untimed warm-up and three timed repetitions per part, median reported, each part in its own process; method in [measurement.md](https://recursiveai.org/onediff/docs/measurement/). The peak is the operating system's maximum resident set size. Budgets are targets of this release. | Row | Budget | x64 | arm64 | Verdict | |---|---|---|---|---| | PDF pair: read both, compare, mark the new pages | 2 s | 841 ms, peak 478 MB | 726 ms, peak 485 MB | pass | | The same pair to a rendered redline, inline | 2 s and peak 1 GB | 1,178 ms, peak 639 MB | 1,197 ms, peak 608 MB | pass | | The same, side by side | information | 1,717 ms, peak 718 MB | 1,738 ms, peak 686 MB | — | | Memory above the inputs per page: markup / rendered inline / side by side | information | 4.1 / 5.7 / 6.5 MB | 4.1 / 5.3 / 6.1 MB | — | | Scan pair: compare + mark after OCR (20-pair slice of `scan-v1`, about 6 pages per pair) | OCR time + 1 s | at most 145 ms (median 6 ms); OCR 1.7 s per pair (332 ms per page) | at most 122 ms (median 5 ms); OCR 9.2 s per pair (1.7 s per page) | pass | The scan OCR time is the recognition engine's, not ours; on the Mac it was measured on a loaded machine. ## Platforms | Check | x64 | arm64 | Verdict | |---|---|---|---| | Whole-solution tests (all test projects, Release) | 864 tests: 848 passed, 16 skipped, 0 failed | 864 tests: 846 passed, 18 skipped, 0 failed | pass | | Build warnings with warnings as errors | 0 | 0 | pass | | NativeAOT smoke app (every package's main path), `-warnaserror` | 0 warnings; runs and prints `ok`; 55.2 MB | 0 warnings; runs and prints `ok`; 53.8 MB | pass | | NativeAOT `onediff` tool, `-warnaserror` | 0 warnings; `diff` exits 1 and prints the change; 48.3 MB | the same; 47.0 MB | pass | | Pack dry run: exactly `RecursiveAI.OneDiff`, `.Documents`, `.Ocr`, `.Pdf`, `.Tool`, each with the licences, notices and readme | 5 packages, set matches | 5 packages, set matches | pass | The test run and the pack dry run were repeated after the adapter packages moved to the renamed document library: the same counts (864 tests, 0 failed; 5 packages) on both machines. Skipped tests are those that need a native host or a file set that is not present (including the pinned failing read-back test above). Browser WebAssembly (headless Chrome 155, x64): Core 361 tests, 353 passed, 0 failed, 8 skipped; `.Documents` 200 (197 passed, 3 skipped); `.Pdf` 135 (119 passed, 16 skipped); `.Ocr` 18 (8 passed, 10 skipped); skips are tests that need files, threads, processes or AES, marked native. The `.Ocr` skips include every test that reads a scan: under WebAssembly the image library (OneImage 1.10.5) throws a `TypeLoadException` while rasterizing, so scans are not supported in the browser until a OneImage fix (expected in OneImage 1.11). ## DOCX comparators on docx-v1 (information; not part of this release) Set: `docx-v1`, sha256 `62b43875b9746cf41e59fafefa643e3ddf02c3301c490a6cf1eae7ce6697f91e`: the 1,050 evaluation pairs of the synthetic set rendered to Word files in six looks. The free comparison tools were run as released binaries through their public interfaces, one run each, with default settings except where noted in [`results/docx-v1-comparators.md`](https://github.com/recursive-ai-oss/onediff-csharp/blob/6443915c4136cc8c5a49fa2543322011665f309e/docs/results/docx-v1-comparators.md) (Docxodus 15.0.0 package, OfficeIMO.Word 3.5.3, WmlComparer through python-redlines 2.0.0, LibreOffice 26.8.2.2 headless). **Indicative**: one run each, on a shared machine. Bootstrap 95% intervals over documents (1,000 resamples). **OneDiff's own rows on this set: pending: the DOCX package is not in this release.** The engine's tracked-changes DOCX output is not scored here. | Tool | Pairs run | Accept-all = new | Reject-all = old | |---|---|---|---| | Docxodus | 1,050 / 1,050 | 1,050 | 1,050 | | OfficeIMO | 1,050 / 1,050 | 1,050 | 958 | | WmlComparer | 1,050 / 1,050 | 1,050 | 1,050 | | LibreOffice | 884 / 1,050 (166 files of one look did not load) | 883 | 763 | | OneDiff | pending: the DOCX package is not in this release | pending | pending | | Row | Docxodus | OfficeIMO | WmlComparer | LibreOffice | OneDiff | |---|---|---|---|---|---| | align F1, edited | .952 [.943, .960] | .934 [.923, .944] | .928 [.916, .939] | .802 [.771, .829] | pending | | align F1, moved | .919 [.885, .948] | 0 | 0 | 0 | pending | | align F1, split | .992 [.972, 1.000] | 0 | 0 | 0 | pending | | align F1, merged | .989 [.970, 1.000] | 0 | 0 | 0 | pending | | align F1, unrelated | .851 [.820, .878] | .731 [.694, .766] | .740 [.703, .776] | .094 [.082, .106] | pending | | boundary F1 | .689 [.668, .710] | .337 [.312, .364] | .561 [.526, .594] | .435 [.408, .462] | pending | | fragmentation: changes per changed sentence | 1.144 [1.115, 1.177] | 1.000 | 1.060 [1.048, 1.073] | 1.050 [1.038, 1.064] | pending | | fragmentation: changes per gold unit | 1.405 [1.362, 1.452] | 1.215 [1.184, 1.249] | 1.314 [1.278, 1.351] | 1.350 [1.308, 1.396] | pending | | missed changed sentences | .072 [.063, .082] | .050 [.041, .059] | .043 [.035, .051] | .221 [.182, .261] | pending | The comparators' zeros for moved, split and merged are not misreads: only Docxodus writes move markers, and none of the others can express a link between paragraphs. OfficeIMO's fragmentation of 1.000 is a paragraph per change (its boundary F1 is .337). Speed of the comparators on joined pairs of about 5, 40 and 120 pages is in the linked file (indicative); the DOCX speed rows of OneDiff are pending with the package. Rows where OneDiff's DOCX output and the free tools are compared (strictly better, the ceiling clause, the blind-preference rows) are not published in this release. ## Failures, rows below their floor, and gaps - **PDF source, changes per gold unit: floor re-based** to 1.174 [1.133, 1.219] from 1.135, because the earlier value counted wrong matches of short equal cells across table rows. An inserted or deleted table row now costs one change per cell; making it one change is planned. Every other `pdf-v1` row is at or above its floor. - **At the floor within rounding:** PDF/PDF moved align F1 (.8429 against .843). - **Scan limits:** scan boundary F1 (.237), split and merge recovery (F1 .055 and .109) and moved precision are limited by recognition errors and by paragraphs the recognition engine merges; the flags (precision .93, recall .62) mark rather than hide them. Latin-script scans only. - **Known limit:** one 29.7 MB pair exceeds the renderer's display-list limit; the rendered redline gives a clear error for it. It is a limit, not a failure of the gates above. - **Text-core rows that lose:** none. Patch make on the medium size is slower than diff-match-patch's by 1.13× on x64 and 1.02× on arm64, inside the bar (geomean .85 and .83); peak memory on the largest line and word inputs is above DiffPlex's (5.7 against 5.3 MB and 14.7 against 12.7 MB), and no bar is set on it. - **Not run in this release:** DOCX engine rows and DOCX speed rows against the free DOCX tools, blind-preference rows, grid and slide rows, the model rows, and a WebAssembly timing row; the packages that produce them ship later. There is no managed comparator for PDF, scan, merge or document speed, so those rows are budgets only. - **Platform coverage:** the browser suites ran on x64 only; native AOT, the whole-solution tests and the pack dry run ran on both machines. The scan-time and OCR rows were taken on a 20-pair slice.  # Measuring OneDiff How the exactness gates and the speed rows are reproduced. Everything here is tooling: it lives in `tools/` and `bench/`, nothing in it ships in a package, and the comparators and reference tools it uses are only ever executed or called through their public interfaces. ## Reference tools `tools/install-oracles.sh [ROOT]` installs pinned released binaries under `ROOT/oracles` (default `$ONEDIFF_HOME`, else `~/.cache/onediff`) without touching the system: | Tool | Version | Source | |---|---|---| | git | 2.51.2 | conda-forge binary through a user-local micromamba | | GNU diffutils (diff, diff3) | 3.12 | same | | GNU patch | 2.8 | same | | diff-match-patch (Python) | 20241021 | PyPI wheel (`--only-binary`) | The script prints each tool's own version line and exits non-zero if one differs from its pin. Linux x64 and macOS arm64 are supported. The GPL tools are run as processes by the harness and the benchmarks and are never referenced from a package. ## Corpora `tools/fetch-corpora.py [--root DIR] [repos] [wiki] [synth]` builds the text and merge corpora under `DIR` (default `$ONEDIFF_CORPORA`, else `$ONEDIFF_HOME/corpora`). Nothing is committed. - `text/<source>/<n>/{old,new}`: version pairs from the history of six permissively licensed repositories, from Wikipedia revisions (seven languages) and from a seeded set of edge cases (emoji and joiners, combining marks, CRLF/LF mixes, missing final newline, a 1.2 MB single line, a 150k-line file, empty and identical sides). - `merge/<repo>/<n>/{base,ours,theirs}`: three-way triples from real merges that changed the same file on both sides. - A licence or terms file beside every source, and `manifest.json` with a SHA-256 for every file. `tools/corpora-pins.json` pins every repository to a commit and every Wikipedia page to a list of revision ids, so a fresh fetch yields the same files. The manifest's own SHA-256 is the corpus identity; it is printed at the end of the fetch and recorded in every gate report. ## Exactness gates `tools/OneDiff.Oracles` is a console tool that runs the reference tools as black boxes over the corpus: ``` dotnet run -c Release --project tools/OneDiff.Oracles -- run \ --corpus ~/.cache/onediff/corpora --oracles ~/.cache/onediff/oracles --out out --cores 8 ``` | Gate | Checks | |---|---| | `min-edit` | our Myers edit distance on lines equals `diff --minimal`'s (lines end at `\n`, keeping their terminator, as GNU diff cuts them) | | `validity` | for every algorithm and for line, word and character tokens: the script is well formed, equal runs are equal, applying it gives new and undoing it gives old, and no algorithm beats Myers | | `roundtrip-unified`, `roundtrip-dmp` | our patch, applied by our code, reproduces new | | `accept-gnu-patch`, `accept-git-apply` | our unified output applied by GNU patch and by `git apply` reproduces new | | `dmp-interop` | our diff-match-patch patch text applies in the Python package and the package's patch text applies in ours | | `merge-file:merge`, `:diff3`, `:zdiff3` | our three-way merge equals `git merge-file -p` byte for byte in each conflict style, with the conflict count; each mismatch keeps a unified diff in `merge-mismatch/` | The patch and merge gates call the library through one file, `tools/OneDiff.Oracles/Seam.cs`. `--seam oracle` answers from the oracles themselves, so every gate must pass against it; that tests the gate plumbing. Cases run in parallel (`--cores`), are listed in ordinal order, and each finished case is appended to `results-<seam>.jsonl`; rerunning with the same `--out` skips them (`--fresh` starts over). `report.json` has no timings and is byte-identical across runs. Oracle timeouts and inputs too large for the Python package are counted as skips, never as passes. Exit code 1 means some gate has failures. ## Benchmarks `bench/OneDiff.Bench` holds the BenchmarkDotNet suites. The comparators, DiffPlex 1.9.0 and DiffMatchPatch 5.0.0 (the diff-match-patch authors' C# package), are restored for this project only. Inputs are corpus pairs of about 2 KB, 40 KB and 400 KB (the nearest pair available, named in the report), read at run time. | Area | Ours | Comparators | |---|---|---| | lines | `Diff.Lines` per algorithm | DiffPlex line differ; diff-match-patch line mode | | words | `Diff.Words` per algorithm | DiffPlex word differ | | chars | `Diff.Characters` per algorithm | DiffPlex character differ; diff-match-patch `diff_main` | | cleanup | semantic and efficiency | diff-match-patch cleanup | | patch | make, toText, fromText, apply | diff-match-patch patches | Settings, so the work is the same on both sides: - Everything is single-threaded, workstation non-concurrent GC. - No cleanup in the diff rows: each implementation returns its raw edit script. - diff-match-patch runs with `Diff_Timeout = 0` (no time limit), so it finds the full diff. Its line mode maps each distinct line to one character, diffs the characters and reads the result in lines; the package keeps its own line mapping non-public, so the mapping is done in the benchmark, inside the timed work. - DiffPlex's word differ cuts words at the separators it is given (spaces, line ends and common punctuation), so the word rows do not see identical token streams. `info` prints the edit distance each implementation finds on each input; lines and characters agree between DiffPlex and diff-match-patch. - Cleanup rows start each call from a copy of the same diff, and the copy is timed on both sides. `bench/run-bench.sh OUT [BenchmarkDotNet args]` runs everything: the suites (with `ONEDIFF_BENCH_MODE=smoke` for a short indicative pass), the input and edit-distance listing, peak working set per row (one cold run per child process, reported above a load-only baseline), and `results-table.md`: per size, ours against each comparator with time ratio, allocations and allocation ratio, a geometric mean per area and comparator, and the verdict against the bars (geomean at most 1.0, no row above 1.25). `bench/record-run.sh` writes the CPU model, core count, instruction-set flags and the load average before, during and after into the same folder. Timings from a machine that was doing other work are not results. ## Restoring the sister-library previews `RecursiveAI.OneDiff.Pdf` builds on a preview of `RecursiveAI.OnePdf` that is not on nuget.org yet. Until it is, restore needs one extra package source that is kept out of the repository: 1. Put the preview `.nupkg` files (`RecursiveAI.OnePdf`, `.Layout` and `.Fonts` at the version in `Directory.Packages.props`) in a folder and point `ONEDIFF_PREVIEW_FEED` at it. 2. Copy `docs/nuget-local.config.example` to `NuGet.Config` at the repository root. The file is not tracked, so no machine path ever lands in a commit. It maps only the OnePDF package ids to that folder and everything else to nuget.org; the mapping also keeps central package management from warning about several sources. 3. `dotnet restore` as usual. Without the file, restore of the PDF projects fails with an unresolved package version; every other project restores from nuget.org alone. Hosted CI cannot restore the PDF projects until the version is published. ## Gold sets The document engine is measured against labelled pairs. Three kinds of data exist, and each has a fixed role: **synthetic pairs with exact labels** (the scored set), **the same pairs rendered to PDF plus real version pairs** (the PDF gate set and the exact gates), and **real pairs without labels** (invertibility, fallback and re-extraction only). Nothing in this section is committed except the tools; data lives under `~/onediff/eval/` (evaluation) and is regenerated from seeds and pins. ### Synthetic pairs `tools/OneDiff.GoldSets generate --out DIR [--role eval|dev|train] [--seed N] [--docs-per-lang N] [--verify]` writes labelled pairs. - **Documents** are contract-like and written for the generator, in seven languages (English, German, French, Spanish, Chinese, Japanese, Arabic): a title, a preamble, three recitals, a definitions article, five to eight articles of numbered clauses (one to three sentences each), a fee table and a closing line. Sentences come from twelve templates per language with two phrasings each; amounts, durations, dates, percentages, jurisdictions, one defined term and cross-references (`Section 3.2`) are filled in from small value lists. Nothing is copied from any real document. - **Edits** (twenty types) each produce labelled change units: `term-swap`, `value-amount`, `value-date`, `value-duration`, `value-percentage`, `clause-rewrite` (the same content in the other phrasing), `sentence-insert`, `sentence-delete`, `para-insert`, `para-delete`, `move`, `move-edit` (moved and changed), `cascade` (a clause inserted before a referenced one, so labels and cross-references renumber), `term-rename` (the defined term changes everywhere), `split`, `merge`, `format` (bold on a word), `table-cell`, `table-row-insert`, `table-row-delete`. Renumbering that follows from any edit is labelled too (`renumber` units and `cascade-reference` units, group `cascade`). - **Pairs.** Per base document: one mixed pair (`-m`, four to eight edits) and four single-type pairs (`-s1` to `-s4`, one to three edits of one type; the types rotate so each is covered in every language). A pair never edits the same sentence twice, so units never overlap. - **Split.** Per language 60 base documents are shuffled with a seeded generator and cut into 30 `eval`, 6 `dev` and 24 `train` documents. All pairs of a base document share its role. The split does not depend on which role is written: `split.json` lists every document with its role and the SHA-256 of its base text. - **Files.** `pairs/<id>.docs.json` (old and new document as compare-model nodes: kind, parent, text, attributes, formatting runs; a node's id is its position) and `pairs/<id>.labels.json`; `split.json`; `manifest.json` (every file's SHA-256, counts per language and edit type, the SHA-256 of `split.json`); `manifest.sha256`, the **set hash**. The same seed gives the same bytes. - **Labels.** `links`: content blocks (paragraphs and list items) by node id, kind `match`, `move`, `split` or `merge`; `deleted` and `inserted` content blocks; `units`: each human-meaningful change with its `type`, `group` (`rename`, `cascade`), `typed` value (kinds `amount`, `date`, `duration`, `percentage`, `term`, `modal`/`adverb`/`jurisdiction` swaps, `reference`, `number`; values in a language-neutral canonical form such as `USD 12500`, `30 d`, `2027-03-01`) and its character spans on each side; `sentences`: the changed gold sentences. A unit without spans (`split`, `merge`, `renumber`) is structural. - **Sanity gate.** `--verify` runs every pair through the current pipeline with nothing hidden and requires `Verify` to pass at fallback level 0. The generator also refuses a pair in which a difference between the documents is not covered by a unit. ### Keeping evaluation out of training Evaluation data is written only under `~/onediff/eval/`; training data under `~/onediff/train/`, and no tool reads across. `generate --role train` never writes eval pairs, and the roles of a seed are fixed by the shuffle above, so a training set cut from `--role train` of the **same seed** is disjoint from the eval set by construction. `check-split --a EVALDIR --b TRAINDIR` proves it for any two directories: it compares the SHA-256 of every base document and exits 1 on a shared one. Distillation and head training load `train` and `dev` only. Real pairs and the PDF renderings of eval pairs are eval. The eval manifest (set hash) is recorded before any model sees training data. ### The PDF gate set `tools/OneDiff.GoldSets.Pdf render --data DIR [--real DIR] --out PDFDIR` renders every synthetic pair (and every real pair) twice, old and new, with OnePDF's layout engine. The layout is chosen from the pair id (four variants: a single justified, hyphenated column on A5; two columns on A5 with a header; one wide-margin column on A4 with a header; two narrow columns on A5 with a header), all with a "Page n / N" footer, tables, lists, page- and column-spanning paragraphs, soft-hyphenated line ends, CJK (Noto Sans CJK) and Arabic (Noto Sans Arabic, right to left). The file id is derived from the file name, so equal input gives equal bytes. After rendering, the text is extracted again with OnePDF and every content block is looked up in order; `pdf/<id>.blocks.json` keeps node id, label, text and page per block, and the manifest reports how many blocks were located. A block that is not found is a limit of extraction (a title word split across lines, a percent sign in right-to-left text), not a missing label. The gold node ids of a pair are the ids of the source documents; a PDF source numbers its own blocks, so scoring a PDF comparison maps blocks by text through `blocks.json`. ### Real pairs `tools/fetch-real-pairs.py` fetches version pairs into `~/onediff/eval/real-pairs-v1` and records everything in `tools/real-pairs-pins.json` (URLs, repository commits, EDGAR documents, each with the SHA-256 it had; a changed document stops the fetch). Real pairs carry no labels: they exercise invertibility, fallback and re-extraction. Sources and terms are in the script's header; nothing fetched is committed. SEC requests are paced at 4 per second under a declared user agent (`ONEDIFF_SEC_CONTACT`). ## Scoring `tools/OneDiff.GoldSets score --data DIR --out FILE [--pipeline NAME]` runs the reference pipeline with nothing hidden (`CompareSettings.Strict`, no throwing), or reads `Comparison` JSON from `--cmp DIR` (`<id>.cmp.json`), checks every file against the manifest, scores each pair and writes rows. A row is a metric on a scope (`all`, `lang:xx`, `mixed`, `edit:<type>`) with its value, a bootstrap 95% interval, the number of documents and pairs, the set hash and the pipeline name. - **Content blocks** are paragraphs and list items; table, row, cell and section nodes carry no text and are not scored. - **Alignment F1 per link type.** A link is typed from its alignment kind and texts: `same` (a match with equal text), `edited` (a match with different text), `moved`, `split`, `merged`; a content block with no link is `unrelated` (deleted or inserted). A predicted link is correct when its type and its sets of old and new node ids equal a gold link's. Precision, recall and F1 per type, from counts summed over documents. - **Boundary F1.** A change unit's spans are trimmed of white space; the boundaries are the (side, node, offset) of every span start and end. Predicted spans come from the engine's changes of all kinds except attribute, split and merge (those have no text span); a side of a change is mapped to its nodes' text. Boundary precision, recall and F1 are over exact matches. - **Fragmentation.** Two counts are kept apart, because a display region can be as large as one likes and still look like "one per sentence". - A change's **text extent** is the union of its sides' trimmed spans. A move change spans its whole node, so it has an extent only for a gold move unit; the edits inside a moved node are changes of their own. Attribute, split and merge changes have none. - A **changed gold sentence** is a sentence of the new document that carries a text change (of the old document, for deleted text). A gold unit's target is its spans. - `frag.sentence` is the mean, over changed sentences touched by at least one change, of the number of **changes** whose extent overlaps the sentence; `frag.unit` is the same per gold unit (the changes overlapping any of its spans; a gold move unit counts move changes). 1.0 means one change per sentence or unit; more means a change is shown in pieces. This is the fragmentation row. - `regions.sentence` and `regions.unit` are the same means over **display regions** (change groups) that hold a counted change. They say how the pieces are presented, never how many there are: a whole-paragraph region holding five changes is five changes and one region. Read `frag` for unit quality and `regions` for display. - `missed.sentence` is the share of changed sentences no change touches. - Fragmentation is meaningful only beside boundary precision: one unit spanning a whole paragraph also has one change per sentence, and pays in `boundary.P`. No row is read alone. - **Pairs with no difference.** A pair whose two documents are identical (the generator can undo a defined-term rename with a second one) has nothing to compare, whatever its unit labels say. Its unit and sentence labels are left out of the boundary and fragmentation counts and `pairs.nodiff` reports how many pairs were left out. Alignment rows still count them. - **Aggregation.** Counts are summed per base document (all pairs of the document in the scope); documents are the resampling unit. The interval is the percentile bootstrap (2.5% and 97.5%) of the metric recomputed on 10,000 resamples of the documents with replacement, seed 20261104. The tool also prints the per-document distribution (minimum, median, 90th percentile, maximum) of boundary F1, changes per changed sentence and regions per changed sentence. `--resamples N` changes the resample count (0 skips the intervals, for sweeps that compare point values); every reported row uses 10,000. `<out>.missed.txt` lists, per pair, the changed sentences no change touched. A metric whose denominator is zero in more than 10% of resamples has no interval. - **Verification** is scored too: `verify.pass` and `verify.fallback` are the shares of pairs that passed and that needed a coarser grouping. ### Semantic rows Scored on the engine's own comparisons of the gold documents (`CompareSettings.Strict` with `CollapseCascades` and `TypedChanges` on, no throwing; the gold nodes are the engine's nodes), so they are not computed for PDF or DOCX renderings. Definitions (docs/semantics.md has the rules being measured): - **Mapping.** A gold unit is mapped to the change whose text extent overlaps its old spans most, and to the change that does for its new spans (one change for a replacement; a deleted and an inserted table cell are two). Moves map only to move changes and nothing else maps to them. A `renumber` unit has no spans; it maps to the attribute change that moved that list label (from and to equal, ignoring a closing full stop). - **`type.acc.<kind>`.** Over gold units with a typed value of the kind that were mapped: the share whose mapped change(s) have the engine's type for it. Kinds and types: `amount` Amount, `date` Date, `duration` Duration, `percentage` Percentage, `term` DefinedTerm, `reference` Reference, `number` (the label of a `renumber`) Numbering. `modal`, `adverb` and `jurisdiction` swaps are plain substantive text changes with no type and are not scored. `type.acc.all` sums the kinds; `type.coverage` is the share of typed units (those kinds) that were mapped to any change. - **`value.acc.<kind>`.** Over the same mapped units: the share whose old value from the old-side change and new value from the new-side change equal the gold value exactly. Gold values are canonical strings (`USD 25000`, `30 d`, `2027-03-01`, `Software`); a gold reference is the text of the reference (`Section 8.4`) and is compared by its number, a gold label by itself without a closing full stop. - **`cascade.P` / `cascade.R`, `rename.P` / `rename.R`.** A gold group is the set of units with that `group` in a pair. One of the engine's groups of that kind *finds* it when the units that its changes are mapped to are exactly the gold set. Precision is found groups over the engine's groups of the kind, recall found groups over gold groups. - **`substantive.P` / `substantive.R`.** Gold substantive is every unit type except `renumber`, `cascade-reference`, `format`, `split` and `merge` (list renumbering and its references, formatting and paragraph structure; split and merge have no spans and are not scored). A unit is predicted substantive when a change mapped to it is. A unit that maps to no change counts as not predicted. - **Pairs with two renames.** In some pairs the generator renamed the term twice and the second rename overwrote the pieces of the first; the first rename's units then describe text that is not in the documents. Their rename units are left out of every semantic row (`pairs.renamechain` counts the pairs: 6 of 210 on dev). This is a label artefact; synth-v2 does not make it. - Pairs with identical documents are excluded as for the other rows. The bootstrap is the same (documents, 10,000 resamples, seed 20261104). - `<out>.semantic.txt` lists every miss with the pair id. ## DOCX gold set and comparison tools The DOCX set is the evaluation split of the synthetic pairs written as Word files, so the labels carry over exactly. Everything here is indicative until a final run is published (`docs/results/docx-v1-comparators.md`). ### Rendering and block maps `tools/OneDiff.GoldSets.Docx render --data DIR --out DIR` writes `docx/<id>.{old,new}.docx`, `<id>.blocks.json` and a manifest whose hash is the set identity. Packages are assembled from OOXML parts and opened and read back through OneDocument, which is also how list labels are checked (`labelsOk` equals `labelsTotal` on every side). Six looks, picked by a hash of the pair id, so each language and edit type appears in each: three plain looks (different fonts, page set-up and table styles), one with bookmarks and cross-reference fields, one with a footnote and a comment on the title, one with a content control around the title. Every look has numbering, a table with a header row and a merged-cell signature box, a header with a picture and a footer with page-number fields. A block map lists, per side, the gold content blocks in order (`node`, `kind`, `label`, `text`, and for PDFs the page). PDF and DOCX renderings write the same shape. The scorer maps a source's content nodes onto gold blocks by the longest common subsequence of equal texts after dropping white space, soft hyphens and direction marks, so order is kept and repeated texts pair up in order; a block's list label may prefix the text (a PDF prints it). Offsets are mapped by counting the characters kept. A source node that matches nothing gets an id past the gold nodes, so lookups never fail and the node counts as unmatched in the printed `block map: matched/total` line. - `score --cmp DIR --blocks DIR` maps comparisons made from rendered files (read as `<id>.cmp.json`); `OneDiff.GoldSets.Pdf compare --pdf DIR --out DIR [--every N]` makes them from PDFs. - `score --changes DIR --blocks DIR` maps another tool's output, read as `<id>.changes.json` (schema `onediff.changes/1`: two line views and the changes as line intervals). A missing file scores as an empty result and is counted as "pairs without output". `--ids FILE` limits the pairs. ### Tracked-changes outputs `OneDiff.GoldSets.Docx read --tool T --tracked DIR --docx DIR --out DIR` reads each tool's DOCX through OneDocument's markup and revision model, writes `<id>.changes.json`, and runs accept-all and reject-all on copies, comparing the paragraph sequence with the new and old input (`roundtrip.jsonl`). The two views are the document with every revision rejected and with every revision accepted, one line per non-empty paragraph; a removed paragraph mark joins a paragraph to the next (ECMA-376 §17.13.5). A change is a run of adjacent revisions in one paragraph with no non-blank unchanged text between them (white space between two revisions does not split a change); the two halves of a move are one change, paired by the name on their range markers; formatting revisions are not changes. Every change is its own display region, so `regions.*` equal `frag.*` for tool outputs. One package quirk is repaired before reading: some tools type every `.xml` part through one default content type and give the main part no override; the reader adds the override and changes nothing else. ### Comparison tools Run only as released binaries, from the harness in `tools/OneDiff.Comparators` (in-process .NET packages), `tools/comparators/wmlcomparer.py` (a Python package that carries the engine as a compiled binary; install with `--only-binary` into a virtual environment) and `tools/comparators/libreoffice_compare.py` (headless office through its UNO bridge, run with the Python shipped in the office's program folder). `tools/install-libreoffice.sh` installs LibreOffice 26.8.2 from the release tarball (sha256 checked) under the oracle root, Linux x64 or macOS arm64; the office is only executed. Pins and settings are in `docs/results/docx-v1-comparators.md`. Each runner records one line per pair (`runs.jsonl`: ok, milliseconds, output size, error) and skips recorded pairs, so a run resumes. `tools/score-docx-comparators.sh` scores each tool on all pairs, on completed pairs only and per look; `tools/summarize-docx-comparators.py` makes the tables. The engine's own row is scored the same way once its Word package writes comparisons: `score --cmp DIR --blocks docx/` and the name `onediff` in the script. ### Real DOCX pairs `tools/fetch-real-pairs.py --only docx --root DIR` fetches eight pairs of related Bonterms templates (CC BY 4.0) pinned in `tools/real-pairs-pins.json` with their SHA-256. They carry no labels: the gates are that each tool runs, the output opens, and accept-all and reject-all reproduce the inputs. ### DOCX speed `bench/OneDiff.Bench.Docx` times document in, tracked-changes document out, in process and single threaded, for the in-process tools, on joined gold-set pairs of about 5, 40 and 120 pages (`OneDiff.GoldSets.Docx bench-pairs --data DIR --out DIR --lang en --sizes 1,8,24`). One command runs it with the machine record (CPU, cores, ISA flags, load before and after): ``` dotnet build bench/OneDiff.Bench.Docx -c Release ONEDIFF_DOCX_BENCH=<bench-pairs folder> bench/run-docx-bench.sh OUT ``` The engine's row is one more `[Benchmark]` method in `DocxBench` with the same shape (open both, compare, save). The out-of-process tools' wall time per pair is in their `runs.jsonl` and is information only, since it includes process start. ### Combine and chain The combine set is built by `OneDiff.GoldSets generate-combine --out DIR` (seed 20261104, same shuffle as the synthetic set, so the originals are its eval documents; nothing in it trains a model) and scored by `score-combine --data DIR --out FILE [--jobs N] [--resamples N] [--explain CASE]`. `tools/run-combine-gate.sh set|sweep OUT` runs the whole gate on the worker through the queue. **The set.** `combine-v1`, sha256 `5c55ebb558a267e2264c7f84401b79d7d9208ed29bcb29157192abbb92def406`: for each of the 210 eval originals (30 per language, seven languages), three cases with 2, 3 and 4 reviewers. A reviewer makes one to three edits of the span-bearing types (term swap, value amount/date/duration/percentage, clause rewrite, sentence delete, paragraph delete). A case's plan is `shared` (some reviewers replay one random stream, so they make the same edits), `conflict` (some reviewers replay the stream that picks the target but draw their own new value, so they edit the same place differently), `shared+conflict` (four reviewers: two pairs) or `independent`. Insertions are left out of the scored rows (their place in the original is not a span) and are covered by the property sweep. **Gold.** Derived from the labels, not from the plan, so a replay that diverged is still exact. A gold edit is a unit's old spans with its new text; reviewers who made exactly that edit form its reviewer set (shared when there are two or more). The spans are narrowed to the characters that really change (a character diff of the old span and the new text, each range widened to whole words except in Han and kana). Two gold edits of different reviewers conflict by the library's overlap rule (shared character, a point strictly inside a range, the same point) and are chained into gold conflict components. The reference and label changes that a deleted paragraph causes are not labelled; they are left out of the predicted side too. **Rows** (documents are the resampling unit: all cases of one original together): - *attribution accuracy*: of the gold edits, the share for which every predicted edit that overlaps it and names one of its reviewers names exactly its reviewer set; - *shared-edit merge recall / precision*: the same test over gold edits with two or more reviewers / the share of predicted edits with two or more reviewers that equal the reviewer set of a gold shared edit they overlap; - *conflict recall*: the share of gold conflict components touched by a predicted conflict at two or more of their edits; *conflict precision*: the share of predicted conflicts (not made only of cascade changes) that touch two gold edits of one component; - *invertible rate*: the share of cases whose combination verifies (`Check()` empty, no error diagnostic) at fallback level 0 with every reviewer's comparison at level 0. **Gates.** The seeded property sweeps are the tests `RandomReviewersAlwaysVerify` (1 to 4 reviewers per random document tree, with shared mutations replayed from one stream; `ONEDIFF_COMBINE_CASES`, `ONEDIFF_COMBINE_SEED`) and `RandomChainsAlwaysVerify` (3 to 5 steps; `ONEDIFF_CHAIN_CASES`, `ONEDIFF_CHAIN_SEED`). The cross-machine digest is `digest-combine --data DIR --count 20`: it hashes the combination JSON and report page of the first 20 cases, twice. Timing is `time-combine`: a combine of 4 reviewers on a 100-page synthetic contract. Results: `results/combine-v1-eval.md`. ## Scan gold set scan-v1 is the digital gold set's Latin-script eval pairs scanned under seeded degradations; `tools/OneDiff.GoldSets.Scan` builds it (`build`, `manifest`), runs the exact gates (`gate`: Verify, fallback, JSON round trip, marks re-read on both sides, determinism, a digest for cross-platform comparison) and times the cost (`time`). Rows come from the digital scorer with `--fuzzy-blocks` (recognized text is matched to the gold blocks by similarity and positions carry over by a page-wide character alignment) plus `noise.changes.share`, `noise.changes.per10kchars` and `ocr.flag.P/R/share`. Method in `ocr.md`, results in `results/scan-v1-eval.md`. ## Budgets `bench/run-budgets.sh OUT` times the document budgets with the machine record (`bench/record-run.sh`: CPU, cores, instruction-set flags, load before, during and after). Each part runs in its own process with `DOTNET_PROCESSOR_COUNT=4`, one untimed warm-up and three timed repetitions, and the table reports the median: - the PDF pair of 100 pages and about 300 changes (the test generator `BudgetTests` in `tests/OneDiff.Pdf.Tests`, seed 2026, written once to `OUT/pair` so that the peak memory of a timed part does not include writing the inputs): read both, compare, mark the new document's pages; - the same pair to a rendered redline, inline and side by side, with the peak working set of each process; - a 100-page DOCX contract with about 300 changes to a tracked-changes DOCX (`TimingTests` in `tests/OneDiff.Word.Tests`; information only); - bytes per page: the peak resident set of the part above that of a process that only loads the same inputs (the `baseline` part), divided by the pages. The peak is the operating system's own maximum resident set size for each process: `/usr/bin/time -l` on macOS, `/usr/bin/time -v` on Linux (or `VmHWM` where GNU time is missing). The .NET process counters are not used, because they read 0 on macOS. A run that cannot measure the peak prints `unmeasured` and exits non-zero rather than print a zero or a negative figure. `ONEDIFF_BUDGET_PAGES` and `ONEDIFF_BUDGET_REPS` shrink a run for a quick check; such runs are indicative. Budgets are proposals: PDF pair 2 s; rendered inline 2 s with a peak of at most 1 GB.  # Building, CI and releases ## Building ``` dotnet build OneDiff.slnx -c Release dotnet test --solution OneDiff.slnx -c Release --no-build ``` The SDK version is pinned in `global.json`. Build output goes outside the checkout: set `ONEDIFF_ARTIFACTS` to choose the folder. Without it, output goes to `onediff/build` under the user's local application data folder (`~/.local/share` on Linux and macOS, `%LOCALAPPDATA%` on Windows), or under the temp folder when there is no such folder. ## Sister-library previews The released packages (`RecursiveAI.OneDiff`, `.Documents`, `.Pdf`, `.Ocr` and `.Tool`) depend only on packages published on nuget.org: OneFont 1.4.0, OnePDF 1.18.0, OneImage 1.10.5, OneRender 1.3.3 and oneocr 1.0.0. A fresh clone with nothing configured builds and tests them. The packages that ship later in 1.x (`.Word`, `.Spreadsheets`, `.Presentations`, `.Rendering`, `.Nlp`, `.All`) and some test and tool projects depend on a prerelease of another Recursive AI library that is not on nuget.org yet (OneDocument with OneArchive and OneCompress, OneNLP). The pins are the prerelease `RecursiveAI.*` entries in `Directory.Packages.props`. Building or testing one of those projects with nothing configured stops at restore with a single message naming the project and the packages, for example: ``` OneDiff.Word needs sister-library previews that are not on nuget.org yet: RecursiveAI.OneDocument 0.2.0-preview.2. Set ONEDIFF_PREVIEW_FEED to a folder or feed that holds them, then restore again (see docs/building.md). ``` **To restore them,** set `ONEDIFF_PREVIEW_FEED` to a folder or feed that holds the `.nupkg` files; `Directory.Build.props` adds it as one more restore source, so no other setup is needed: ``` ONEDIFF_PREVIEW_FEED=/path/to/previews dotnet build OneDiff.slnx -c Release ``` The other ways in: a `NuGet.Config` at the repository root (copy `docs/nuget-local.config.example`; the path is not tracked; the message above is not shown when the file exists and it maps the previews to a folder), or `-p:OneDiffPreviewsConfigured=true` when a source is configured elsewhere, such as the user-level NuGet config. Once those libraries are published, the pins move to the released versions and none of this is needed (the CI selector reads the pins too). ## CI `.github/workflows/ci.yml` runs on every push and pull request: - **build**: build and test on linux-x64, linux-arm64, osx-arm64 and win-x64; publish and run the NativeAOT smoke app (every package's main path: text diff, documents, PDF comparison with on-page markup and rendered output, scans, JSON, HTML, combine) and the NativeAOT `onediff` tool on each with warnings as errors. - **pack**: packs every packable project as a workflow artifact (nothing is pushed anywhere). - **wasm**: publishes the test sources of the core, `.Documents`, `.Pdf` and `.Ocr` as browser WebAssembly apps (`tests/OneDiff.Tests.Wasm`, one per suite) and runs each in headless Chrome. A test that needs files, threads, processes or a native oracle carries `[Trait("Host", "Native")]` (or `Assert.SkipUnless`) and is counted as skipped; the run fails on any other failure. Locally: `tests/OneDiff.Tests.Wasm/run.sh <output-folder> [Core Documents Pdf Ocr]`, which needs the `wasm-tools` workload and `uv` or a Python with `playwright`. - **plan**: decides what the jobs above build, from the preview pins. Hosted runners cannot restore a prerelease that is not on nuget.org, so `plan` runs `tools/ci/select-solution.py`: - With no prerelease pins, or with the repository variable `ONEDIFF_PREVIEW_FEED` set to a feed the runners can reach, everything is built (`OneDiff.slnx`) and the preview config above is written into each job. - Otherwise the jobs build `OneDiff.Hosted.slnf`, generated on the fly: every project of the solution that does not depend, directly or through project references, on a prerelease package. The run prints a notice naming the projects it left out, so a green run never silently covers less than it says. The AOT smoke app runs only if it is among the projects built. ## Releases A package **packs** when its project is under `src/` and `IsPackable` is true: `Directory.Build.props` makes everything non-packable, `src/Directory.Build.props` makes projects under `src/` packable, and a project whose package has no content yet sets `IsPackable` to false until it does. A local `dotnet pack OneDiff.slnx` packs all of them. A package is **published to nuget.org** only when its project also sets `PublishToNuGet` to true (`src/Directory.Build.props` sets it to false). That is the one list of what the release publishes: `RecursiveAI.OneDiff`, `.Documents`, `.Pdf`, `.Ocr` and `.Tool` (the `onediff` command), the packages whose dependencies are all on nuget.org once the sister libraries are released. The release also checks that the packed set is exactly these five and that each carries both licences, the third-party notices and its readme. `EnablePackageValidation` is on in every one of those projects. `.github/workflows/release.yml` builds and tests the solution that restores without prerelease packages, packs each published project, and checks that it produced exactly one package per project marked `PublishToNuGet`. A package joins a release by setting the property; it must not depend on a prerelease. Each package has its own `PACKAGE.md` beside its project; it is the readme shown on nuget.org, so its links are absolute. All packages and the tool share one version, `<Version>` in `Directory.Build.props`. The workflow runs when a `vX.Y.Z[-suffix]` tag is pushed and has three jobs: 1. **verify**: the tag is well formed, its commit is on `main`, CI passed on that commit, and `CHANGELOG.md` has a `## [X.Y.Z]` section. 2. **publish**: every packable project builds the tagged version; the solution is built and tested; the packages are packed from those binaries, attested, and pushed to nuget.org with `--skip-duplicate` (the `.nupkg` files first, then the `.snupkg` symbol packages) using Trusted Publishing, so no API key is stored. 3. **release**: a GitHub release with the changelog section as notes and the packages attached. A version with a suffix (`-preview.1`) is marked as a pre-release. Every action is pinned to a commit. The workflow also honours `ONEDIFF_PREVIEW_FEED` for restoring previews that are not published. ## Packing locally ``` dotnet pack OneDiff.slnx -c Release -o <folder> ``` Packs are built with `Deterministic`, and with `ContinuousIntegrationBuild` when `CI=true`. Every entry of a package (the assemblies, documentation files, licence files and the `.nuspec`) has the same contents on repeated builds. The `.nupkg` and `.snupkg` files as a whole do not hash the same, because NuGet stamps each zip entry with the file's modification time and the pack time. Compare two packages by their entries, not by the file hash: ``` for f in a.nupkg b.nupkg; do unzip -p "$f" | sha256sum; done ```  # Changelog All notable changes to this project are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project follows [Semantic Versioning](https://semver.org/). ## [Unreleased] ## [1.0.1] - 2026-11-05 ### Fixed - `RecursiveAI.OneDiff.Documents`: an unsupported Office input or output (DOCX, XLSX, PPTX) now says that support is coming in a 1.x release and suggests exporting the documents to PDF and comparing the PDFs, instead of naming packages that are not on nuget.org yet; the message starts with a capital letter. - `RecursiveAI.OneDiff.Documents`: attribute changes in HTML and the rendered PDF redline show a readable label and unquoted values (`List number: 1.5 → 1.2`) instead of the internal key; `data-key`, `data-old` and `data-new` and the JSON are unchanged. - `RecursiveAI.OneDiff.Documents`: the cover-page line "Hidden into the list below" reads "Not marked (listed below)". - Documentation: the released sister-library versions, the worst speed row (1.13×), and a known issue: scans cannot be read in browser WebAssembly yet. ## [1.0.0] - 2026-11-05 The first stable release of `RecursiveAI.OneDiff`, `RecursiveAI.OneDiff.Documents`, `RecursiveAI.OneDiff.Pdf`, `RecursiveAI.OneDiff.Ocr` and the `onediff` tool (`RecursiveAI.OneDiff.Tool`). Results: [docs/results.md](https://recursiveai.org/onediff/docs/results/). ### Added - `RecursiveAI.OneDiff`: `TextDiff.Pieces` and `TextDiff.Lines` read a diff as typed runs and lines with line numbers; `Diff.SideBySide` lays a line diff out in aligned rows with the changed words marked; `DiffSettings` (algorithm and work cap) is taken by the `Diff` one-liners and `TextDiff.Diagnostic` reports the cap; `Diff.ApplyUnified`, `ApplyPatchText` and `PatchText` take the apply options, patcher and cancellation token; `TextMerge.Conflicts` and `ItemMerge<T>.Items` expose a merge's conflicts and merged items. - `RecursiveAI.OneDiff`: `HunkResult.Line` and `DmpPatchResult.Location` are `int?` (null when not placed), and `DmpPatcher.Match` became `TryMatch`. - `RecursiveAI.OneDiff`: the `*Using` comparer one-liners (`TextUsing`, `LinesUsing`, `WordsUsing`) and `Diff.Semantic` and `Diff.Efficient` take `DiffSettings`, so the work cap reaches every diff one-liner. `TextDiff.Diagnostic.Work` includes the items of the shared head and tail, so a small real diff no longer reports zero. - `RecursiveAI.OneDiff.Documents`: `Redline.CreateAsync`, `Comparison.OfAsync`, `SaveAsync`, `ComparisonJson.ReadAsync` and `WriteAsync` work on streams that refuse synchronous I/O (for example ASP.NET Core request and response bodies), and every `SaveAs` overload takes a `CancellationToken`. - `RecursiveAI.OneDiff.Documents`: `VerificationResult.VerifiedHere` says whether the completeness checks ran in this process on this value; `ComparisonFormatException` carries the position of a JSON syntax error (`LineNumber`, `BytePositionInLine`, `Path`). - `RecursiveAI.OneDiff.Documents`: `CompareDocument.Origin` holds the bytes a document was read from (and the password), so a PDF is marked from the document itself; `PdfRedline.Auto` chooses from the new document's source format and its origin. - `RecursiveAI.OneDiff.Documents`: `Redline.Register(source, priority)` and `DocumentSources.Register(source, priority)`: a source with a higher priority is asked first, so the order of registration no longer decides which source reads a file. `OcrRedline.Register()` registers the scan source above the PDF source. - `RecursiveAI.OneDiff.Documents`: `RedlineStyle.MarkWholeWords` widens marks inside a word to the whole word, as the page markup and the change list already do. - `RecursiveAI.OneDiff.Documents`: plain text reads leading clause markers (such as `1.1` or `(a)`) as list labels, with the rule the PDF source uses, so renumbering in plain text groups as a cascade the way it does for PDF. - `RecursiveAI.OneDiff.Tool`: `onediff --version`, `--exit-code`, `--whole-words` and `--form`; the tool registers the scan source and reads scans. In packages that are not on nuget.org yet, because they depend on previews of other libraries (`RecursiveAI.OneDiff.Word`, `RecursiveAI.OneDiff.Nlp`): - Word packages: the document library OneDocument (formerly OneOffice) is now referenced as `RecursiveAI.OneDocument*` 0.2.0-preview.2. - `RecursiveAI.OneDiff.Word`: `DocxRedline` writes a comparison as a DOCX with real tracked changes, built on the new document (default) or the old one: `w:ins`/`w:del` for text, paragraphs, rows and cells, `w:moveFrom`/`w:moveTo` with range markers, property changes for paragraphs, tables, rows, cells and sections. Accepting every revision gives the new document and rejecting every one the old. The author is a setting, no date is written unless asked, ids are numbered in document order, and the output is byte-identical from run to run. Format-only edits are run property changes, each change is written as one replaced unit, splits and merges use paragraph marks, changed field results and edited text boxes are tracked in place, and the source reads document properties, the list labels of every story and text boxes through the hub (OneDocument 0.2.0-preview.2). Differences a revision cannot carry are returned in `DocxRedlineResult.NotCarried`. `WordRedline.Register()` makes `Redline.Create` and `SaveAs` use it for `.docx`. See `docs/word.md`. - `RecursiveAI.OneDiff.Word`: `WordDocumentSource` reads DOCX, DOCM, DOTX and DOTM through the OneDocument document hub into the compare model: sections with page setup, headers and footers, paragraphs and list items with resolved numbering labels, tables with merges, footnotes and endnotes, comments, fields (code and result), pictures (content hash, and a perceptual hash for PNG), text boxes, content controls, hyperlinks, resolved run and paragraph formatting, document properties and opaque parts. Anchors name the element and the run of every node and text segment, in the offset space of OneDocument's revision authoring. Tracked changes already in an input are accepted by default, or rejected, or refused (`WordRevisionMode`). Register with `WordRedline.Register()`. See `docs/word.md`. - `RecursiveAI.OneDiff.Word`: `DocxRedline` writes a tracked-changes DOCX for a PDF, a scan, plain text or any other flowing source too. A side that is not a Word file is reconstructed as a DOCX from its compare model (paragraphs, list items with their labels, tables, one header and one footer, character formatting), on the other side's package when that one is a Word file, and labelled as reconstructed in its core properties and by a note in its header (`DocxRedline.ReconstructionNote`); the revisions are then written as for two Word files. The comparison is the one that was verified. See `docs/word.md` and `docs/results/pdf-docx.md`. - `RecursiveAI.OneDiff.Word`: a paragraph whose list is numbered by different definitions in the two packages now gets a property change (the same number can mean different labels in two packages), and blocks that followed a block content control that only the other document has are placed after the copies of what stood inside it, not before. - `RecursiveAI.OneDiff.Nlp`: `NlpValueRecognizer`, an `IValueRecognizer` built on OneNLP's recognizers. It reads written-out English values (`forty-two`, `six percent`, `thirty (30) days`, `two hundred thousand dollars`, `1 year and 6 months`) into the same canonical values as the built-in recognizer, and leaves everything else (other languages, party names, unresolved dates) to it. `CompareSettings.Legal.UseOneNlp()` selects it for one comparison and `OneNlpRecognizers.Register()` for the process. See `docs/nlp.md`. ### Changed - **Breaking:** `ComparisonJson.Read` and `ReadAsync` re-run the completeness checks by default (`verify: true`), so `Verification.Passed` is true only when this process proved it, and a tampered file shows failures. Pass `verify: false` for the cheaper read: the value then reports `VerifiedHere == false` and `Passed == false`, and `Comparison.Check()` returns the failures without changing it (a `Comparison` is immutable; read again with `verify: true` for a proven value). - **Breaking:** a PDF with pages but no text is refused (`InvalidDataException`, `PdfSourceErrors.FailureOf` = `NoTextLayer`, with a message naming `RecursiveAI.OneDiff.Ocr`) instead of being compared as an empty document; a PDF whose pages have no content at all is refused as `Malformed` and described as damaged, with or without the scan source registered. Content OnePDF could not recover becomes the comparison diagnostic `source.<side>.pdf.unrecovered`, which strict settings throw on. A scan in which the OCR engine finds no word carries `source.<side>.ocr.nothingRecognized`, with the same strict behaviour. - **Breaking:** `PdfRedline.Auto` and `SaveAs(".pdf")` no longer consult process-wide state. A comparison read back from JSON has no PDF to mark and says so (use `PdfRedline.Rendered`, or `MarkNewDocument` with the PDF's bytes). - `ItemMerge` is an unsealed base class of `ItemMerge<T>`. - `RecursiveAI.OneDiff`: the line type of `TextDiff.Lines` and `TextDiff.Pieces` is named `DiffRun` (it was `DiffPiece`, which clashed with DiffPlex's type in files that import both during a migration). - `RecursiveAI.OneDiff.Documents`: plain-text comparisons now carry list items, so the JSON of existing text comparisons changes where lines start with clause markers. - Exception messages start with a capital letter. ### Removed - `RecursiveAI.OneDiff.Documents`: `Redline.Compare`, the alias of `Comparison.Of`. Use `Comparison.Of`. ## [0.2.0-preview.1] The first release on nuget.org: `RecursiveAI.OneDiff`, `RecursiveAI.OneDiff.Documents`, `RecursiveAI.OneDiff.Pdf` and `RecursiveAI.OneDiff.Tool`. A first preview, 0.1.0-preview.1, was never published; its contents are included below under that heading. The public API may change before 1.0. ### Added - `RecursiveAI.OneDiff.Documents`: change semantics. `Change` gains `Type` (`ChangeType`: amount, date, duration, percentage, number, party, defined term, reference, numbering, formatting, typography), `OldValue` and `NewValue` (`ResolvedValue`: canonical, language-neutral forms such as `USD 25000`, `2027-03-01`, `30 d`) and `IsSubstantive`. With `CompareSettings.CollapseCascades`, `Stages.Group` lifts a renumbering and the references it caused, and a renamed defined term and its occurrences, into `GroupKind.Cascade` and `GroupKind.Rename` display groups; every underlying change stays in the list. With `TypedChanges`, `Stages.Classify` types and resolves changes. New public seam `IValueRecognizer` (`Stages.Classify(changes, recognizer)`), with `BuiltInValueRecognizer` as the default for the seven measured languages. The JSON gains optional `type`, `oldValue`, `newValue` and `substantive` members on changes (additive in schema version 1); the HTML pages list each cascade and rename group with its occurrences. See `docs/semantics.md`. - `RecursiveAI.OneDiff.Documents`: combine, version chains and the change report (`OneDiff.Documents.Combine`). `Combination.Of(original, copies)` compares every reviewer's copy with the original and combines the results: each change is attributed to the reviewers who made it, identical changes are merged (and each edit is attributed on its own), and changes that cannot all be applied are listed as conflicts with every version shown, never a winner. The changes attributed to a reviewer are exactly that reviewer's verified comparison, checked on every run (`Combination.Check`, with a fallback that lists every change on its own). `ComparisonChain.Of(versions)` verifies every step and the net comparison and says which steps touched each net change (`StepsOf`). `ChangeReport.From(comparison | combination | chain)` gives counts by kind and type, the substantive list with resolved values, conflicts and per-step counts, and renders one self-contained HTML page. `Redline.Combine(original, copies, output)` and `Redline.Chain(versions, output)` are the one-liners; the JSON forms are `onediff.combination` and `onediff.chain` (version 1, schemas in `docs/schema`), which embed unchanged `onediff.comparison` documents. See `docs/combine.md`. - `RecursiveAI.OneDiff.Tool`: `onediff compare` reads PDFs and writes `.pdf` output (the new PDF with the changes marked on its pages, or a rendered redline for other inputs), and takes `-o <output>`; the tool references `RecursiveAI.OneDiff.Pdf`. Rendered redlines draw Latin text only (no script font packages). - `RecursiveAI.OneDiff.Tool`: `onediff combine <original> <copy>... -o <out>` and `onediff chain <v1> <v2>... -o <out>`. - `RecursiveAI.OneDiff.Documents`: `CompareSettings.ValueRecognizer` and `ValueRecognizers.Register` choose the value recognizer of a comparison (settings, else the registered one, else the built-in one); the fallback stages read it from the settings too. Not written to JSON. - `RecursiveAI.OneDiff.Pdf`: the rendered redline. `PdfRedline.Rendered` and `PdfRedline.RenderedSideBySide` (`RenderedPdfRedline`, with `Paper` and `Render`) write a tagged PDF laid out from any comparison, plain text included: the cover page, key, marks, ledger and footer of the HTML redline set on A4 or Letter with a running page number, in any `RedlineStyle`, byte-identical across runs. It depends on `RecursiveAI.OneRender.Pdf`. - `PdfRedline.Auto`, registered for `.pdf` by `PdfRedline.Register()`: original-page markup when the new document is a PDF, a rendered redline otherwise. Both forms can still be named. - Release packaging: the release workflow publishes the projects that set `PublishToNuGet`; a local `dotnet pack` still packs every packable project. Each package has its own readme. The OnePDF and OneFont pins are two properties in `Directory.Packages.props`. - Sample PDFs for every preset and layout, and one original-page markup sample, under `docs/samples`, with a README of the one-line recipes. ### Changed - `PdfRedline.Register()` registers `PdfRedline.Auto` for `.pdf` instead of `PdfRedline.OnPages` alone; a PDF as the new input is marked on its pages as before. ### Not in this release - `RecursiveAI.OneDiff.Word` (DOCX source and tracked-changes writer) and `RecursiveAI.OneDiff.Nlp` (OneNLP value recognizer) are in the repository but not published, because they depend on previews of OneDocument and OneNLP that are not on nuget.org. Change semantics, combine, chains, the change report and the recognizer seam live in `RecursiveAI.OneDiff.Documents` and are in this release. - Scans, spreadsheets, slides, the model and the OneDocument rendering package are not built yet. - `RecursiveAI.OneDiff.Ocr`: `OcrDocumentSource` reads scans into the flowing compare model through the local OCR engine of `RecursiveAI.OneOcr`: images (PNG, JPEG, TIFF, recognized by content) and PDFs whose pages carry no text of their own (`OcrDocumentSource.IsScannedPdf`). Paragraphs, list items (clause numbers included), and running headers and footers (found by their repetition across pages) become nodes whose words are anchored by `AnchorKind.OcrWords`: the page quadrilateral in upright page space and the engine's confidence. `OcrSourceOptions` carries the engine, resolution, limits and layout switches. `OcrRedline.Register()` registers the PDF and scan sources and the on-page markup form; a scan is marked on its own pages (an image on a PDF page made from it) by `PdfRedline`. See docs/ocr.md. - `RecursiveAI.OneDiff.Documents`: possible OCR errors. `Change.OcrConfidence` is the lowest confidence of the recognized words a change's edits touch, and `Change.PossibleOcrError` is true when it is below `CompareSettings.OcrConfidenceThreshold` (default 0.95, chosen on a development scan set). The comparison JSON carries `ocrConfidence` and `possibleOcrError` on changes and `ocrConfidenceThreshold` in the settings when it differs from the default; all three are optional additions to schema version 1. - Gold sets: scan-v1, the Latin-script pairs of the digital gold set scanned under seeded degradations (`tools/OneDiff.GoldSets.Scan`), with exact gates, noise and flag rows and an indicative cost row. See docs/results/scan-v1-eval.md. - `RecursiveAI.OneDiff.Pdf`: the markup writer places marks on `AnchorKind.OcrWords` anchors as it does on glyph quadrilaterals. ## [0.1.0-preview.1] The first preview. Public API may change before 1.0. ### Added #### `RecursiveAI.OneDiff` (text core, no dependency beyond the .NET base class library) - Edit scripts as struct ranges (`Edit`, `EditScript`, `EditScriptBuilder`), `TokenInterner<T>`, and the `Diff` entry points for lines, words, characters and graphemes, with comparer-based variants (ignore case, ignore whitespace). - Myers (linear space, with a work cap and a `DiffDiagnostic` when it is reached), patience and histogram diff over any sequence with an equality comparer. - Line, word, character and grapheme tokenizers; word tokens are script-aware, so Han, Hiragana and similar scripts split per character. - Semantic and efficiency cleanup, with lossless shifting of changes to word, sentence and line boundaries. - Unified diff: make, parse and apply with offset and fuzz, reporting hunks that do not apply. - diff-match-patch patch text: make, parse and apply with fuzzy matching, with UTF-16 or code-point offsets. - Three-way (diff3) merge with `merge`, `diff3` and `zdiff3` conflict styles, resolution options and a structured conflict result. - Terminal and Markdown renderers for line and word diffs. #### `RecursiveAI.OneDiff.Documents` (document comparison engine) - The compare model: block trees with source anchors, formatting runs and normalization folds; `IDocumentSource` with a plain-text source. - The five stages (`Align`, `DiffPairs`, `Group`, `Classify`, `Verify`) over immutable values, an editable `Alignment`, and deterministic block alignment with moves, splits, merges and table-aware matching. - `Comparison` with a change ledger, diagnostics and `Verify`: every comparison is checked for completeness (old plus changes gives new, new minus changes gives old) and falls back stepwise to a coarser complete grouping, or throws in strict mode. - `CompareSettings` and `RedlineStyle` presets, and a versioned JSON form (schema v1) with a published schema. - Self-contained HTML redlines, inline and side by side, with cover page, change bars and footer; the `Redline` statics (`Create`, `Compare`), `Comparison.SaveAs`, `Save` and `ToBytes`, and a registration seam (`DocumentSources.Register`, `Redline.Register`, `RedlineForm`) for adapter packages. #### `RecursiveAI.OneDiff.Pdf` (PDF source and original-page markup) - `RecursiveAI.OneDiff.Pdf`: `PdfDocumentSource` reads digital PDFs into the compare model (paragraphs joined across lines, columns and pages, list items, tables, running headers and footers, formatting, glyph-quad anchors; line breaks, hyphenation and page numbers go to the ledger) and `PdfRedline` marks a comparison on the original pages with real highlight, caret, underline, squiggly and strike-out annotations carrying the old text in their pop-ups. See `docs/pdf.md`. #### Repository - CI on linux-x64, linux-arm64, osx-arm64 and win-x64, a NativeAOT smoke app, and the core tests run as a browser WebAssembly app in headless Chrome. - A release workflow that publishes through nuget.org Trusted Publishing; see `docs/building.md`. ### Changed - Building without `ONEDIFF_ARTIFACTS` set no longer fails on the default output folder.

# onediff docs, all pages (OneDiff 1.0.1) Every page of https://recursiveai.org/onediff/docs/ as markdown, getting started first. Index: https://recursiveai.org/onediff/llms.txt # Getting started with OneDiff Install OneDiff 1.0.1 for .NET 10, diff and merge text, compare two documents into a verified change list, and write a redline as HTML, JSON or PDF. Pure C#, with no native code and no network access. ## Install The packages target `net10.0`, so you need the .NET 10 SDK. ```shell dotnet new console -o OneDiffStart cd OneDiffStart dotnet add package RecursiveAI.OneDiff --version 1.0.1 ``` Five packages are on NuGet at 1.0.1, all on one version: - **`RecursiveAI.OneDiff`**: the text core. Diff, patches and three-way merge, with no dependency beyond the .NET base class library. - **`RecursiveAI.OneDiff.Documents`**: the comparison engine, HTML redlines and the JSON change list. It brings in the core and `RecursiveAI.OneFont`. - **`RecursiveAI.OneDiff.Pdf`**: digital PDFs in; changes marked on the original pages, or a rendered redline PDF. It brings in `.Documents`, `RecursiveAI.OnePdf` and `RecursiveAI.OneRender.Pdf`. - **`RecursiveAI.OneDiff.Ocr`**: scanned PDFs and images, read by a local OCR engine (`RecursiveAI.OneOcr`). - **`RecursiveAI.OneDiff.Tool`**: the `onediff` command-line tool. Word documents are not part of 1.0. In the README’s words: “Coming in later releases: Word documents in and tracked-changes DOCX out (`RecursiveAI.OneDiff.Word`, in this repository and not in the first release), spreadsheets, slides, and the OneNLP value recognizer (`RecursiveAI.OneDiff.Nlp`).” Until then an Office file is refused: ```text NotSupportedException: No document source is registered for this input, which looks like an Office Open XML or other zip-based document. Support for Office documents (DOCX, XLSX and PPTX) is coming in a 1.x release. Meanwhile, export the documents to PDF and compare the PDFs (with RecursiveAI.OneDiff.Pdf). ``` ## Diff, patch and merge text Put this in `Program.cs` and run `dotnet run`: ```csharp using OneDiff; using OneDiff.Merge; string oldText = "Deliver within 30 days.\nPay within 45 days.\n"; string newText = "Deliver within 21 days.\nPay within 45 days.\n"; // The words that changed, as Markdown Console.WriteLine(Diff.ToMarkdownInline(Diff.Words(oldText, newText))); // A unified patch, and applying it string patch = Diff.Unified(oldText, newText, "a/terms.txt", "b/terms.txt"); Console.Write(patch); Console.WriteLine($"applied: {Diff.ApplyUnified(oldText, patch).Text == newText}"); // A three-way merge: edits to different lines merge string baseText = "one\ntwo\nthree\nfour\n"; string ours = "ONE\ntwo\nthree\nfour\n"; string theirs = "one\ntwo\nthree\nFOUR\n"; TextMerge merged = Diff.Merge(baseText, ours, theirs); Console.Write(merged.Text); Console.WriteLine($"conflicts: {merged.HasConflicts}"); ``` ```text Deliver within 3021 days. Pay within 45 days. --- a/terms.txt +++ b/terms.txt @@ -1,2 +1,2 @@ -Deliver within 30 days. +Deliver within 21 days. Pay within 45 days. applied: True ONE two three FOUR conflicts: False ``` Edits to the same or neighboring lines are a conflict, written with conflict markers in git’s `merge`, `diff3` or `zdiff3` style. See [Three-way merge](https://recursiveai.org/onediff/docs/merge/) and [Patches](https://recursiveai.org/onediff/docs/patches/). ## Compare two documents Document comparison is in `.Documents`: ```shell dotnet add package RecursiveAI.OneDiff.Documents --version 1.0.1 ``` Save two versions of a short contract as `v1.txt` and `v2.txt`: ```text 1. Supply. The Supplier shall deliver the goods within 30 days of each order. 2. Payment. The Customer shall pay each invoice within 45 days of its date. ``` In `v2.txt`, change `30 days` to `21 days`. Then: ```csharp using OneDiff.Documents; using OneDiff.Documents.Serialization; Comparison cmp = Comparison.Of("v1.txt", "v2.txt"); foreach (Change c in cmp.Changes.Where(c => c.IsSubstantive)) { Console.WriteLine($"{c.Kind}: {c.Old.Text} -> {c.New.Text}"); if (c.OldValue is { } was && c.NewValue is { } now) Console.WriteLine($" {c.Type}: {was.Canonical} -> {now.Canonical}"); } Console.WriteLine($"complete: {cmp.Verification.Passed}"); // The output form follows the extension cmp.SaveAs("redline.html"); cmp.SaveAs("changes.json"); Comparison again = ComparisonJson.Read(File.ReadAllBytes("changes.json")); Console.WriteLine($"read back: {again.Changes.Length} changes, complete: {again.Verification.Passed}"); ``` ```text Replace: 30 days -> 21 days Duration: 30 d -> 21 d complete: True read back: 1 changes, complete: True ``` The change is typed: a duration, with both values resolved. `complete` is the completeness check every comparison runs: the old document plus the changes gives the new one, and the new one minus the changes gives the old. Reading the JSON back runs the check again, so an edited file fails it. See [The compare model](https://recursiveai.org/onediff/docs/compare-model/) and [Change semantics](https://recursiveai.org/onediff/docs/semantics/). ## Redline PDFs PDF input and output are in `.Pdf`: ```shell dotnet add package RecursiveAI.OneDiff.Pdf --version 1.0.1 ``` With two versions of a digital PDF, `v1.pdf` and `v2.pdf` (here, the two text files above saved as PDF): ```csharp using OneDiff.Documents; using OneDiff.Pdf; PdfRedline.Register(); // once at startup // Highlights on v2.pdf's own pages, with the old text in the pop-ups Redline.Create("v1.pdf", "v2.pdf", "markup.pdf"); // A rendered redline: a new, tagged PDF laid out from the comparison Comparison cmp = Comparison.Of("v1.pdf", "v2.pdf"); cmp.SaveAs("redline.pdf", PdfRedline.Rendered); cmp.SaveAs("side-by-side.html", RedlineForm.HtmlSideBySide); foreach (Change c in cmp.Changes.Where(c => c.IsSubstantive)) Console.WriteLine($"{c.Kind}: {c.Old.Text} -> {c.New.Text}"); ``` ```text Replace: 30 days -> 21 days ``` `markup.pdf` is `v2.pdf` with a highlight and its pop-up note; the page content is untouched. `redline.pdf` is a two-page tagged PDF: a summary of the changes by type, then the text with `30` struck through and `21` inserted. Every look the writers have is in [Samples and redline looks](https://recursiveai.org/onediff/docs/samples/). A PDF with no text layer is refused rather than compared as empty: ```text InvalidDataException: This PDF has no text layer, so it is probably a scan. Reference RecursiveAI.OneDiff.Ocr and call OcrRedline.Register() to read it by OCR. ``` ## Scans Scanned PDFs and images (PNG, JPEG, TIFF) are in `.Ocr`, which reads them with a local OCR engine: ```shell dotnet add package RecursiveAI.OneDiff.Ocr --version 1.0.1 ``` ```csharp using OneDiff.Documents; using OneDiff.Ocr; OcrRedline.Register(); // once at startup; reads scans and digital PDFs Redline.Create("scan-v1.png", "scan-v2.png", "scan-markup.pdf"); foreach (Change c in Comparison.Of("scan-v1.png", "scan-v2.png").Changes) Console.WriteLine($"{c.Kind}: {c.Old.Text} -> {c.New.Text}, " + $"confidence {c.OcrConfidence:0.00}, possible OCR error: {c.PossibleOcrError}"); ``` ```text Replace: 30 days -> 21 days, confidence 0.98, possible OCR error: False ``` A change whose words were read with a confidence below 0.95 (the default threshold) is flagged as a possible recognition error; the flag never hides a change. Scans are Latin-script only, and in the README’s words, “In browser WebAssembly, reading scans with `RecursiveAI.OneDiff.Ocr` fails with a `TypeLoadException` from the image library it uses (OneImage).” See [Scans and images](https://recursiveai.org/onediff/docs/ocr/). ## The onediff tool ```shell dotnet tool install --global RecursiveAI.OneDiff.Tool --version 1.0.1 onediff diff v1.txt v2.txt onediff compare v1.pdf v2.pdf -o markup.pdf ``` ```text --- v1.txt +++ v2.txt @@ -1,3 +1,3 @@ -1. Supply. The Supplier shall deliver the goods within 30 days of each order. +1. Supply. The Supplier shall deliver the goods within 21 days of each order. 2. Payment. The Customer shall pay each invoice within 45 days of its date. ``` `onediff diff` exits 1 when the texts differ, as `diff` does. The other commands are `patch`, `merge`, `combine` (reviewers’ copies of one original) and `chain` (a series of versions); `onediff --help` lists each one’s options. ## Next - [Overview](https://recursiveai.org/onediff/docs/overview/): what OneDiff is, the packages and the happy path. - [Migrating from DiffPlex](https://recursiveai.org/onediff/docs/migrating-from-diffplex/) and [from WmlComparer](https://recursiveai.org/onediff/docs/migrating-from-wmlcomparer/). - [Results](https://recursiveai.org/onediff/docs/results/): every measured result, including the rows that lose. # OneDiff **Text diff and document comparison for .NET.** This project is unaffiliated with any other project named OneDiff, including the Python diffusion-model accelerator of that name. OneDiff is a clean-room diff and comparison library for .NET, in two layers: - **A text diff core** (`RecursiveAI.OneDiff`): Myers, patience and histogram diff over any sequence; line, word, character and grapheme tokenizers; semantic and efficiency cleanup; unified and diff-match-patch patches with fuzzy apply; three-way merge in git's conflict styles. It depends on nothing beyond the .NET base class library. - **Document comparison** (`RecursiveAI.OneDiff.Documents` and the adapters): a compare model with source anchors, alignment that follows moves, splits and merges, typed changes with resolved values (`30 days → 60 days`), renumbering and renamed defined terms as single changes, combining reviewers' copies, and version chains. **PDFs** (digital, and scans through a local OCR engine) are read into the model; the changes are marked on the original pages with real annotations, written as a rendered redline PDF, as HTML (inline or side by side), or as a versioned JSON change list. Every comparison is checked for completeness: old plus the changes gives new, and new minus the changes gives old. If a grouping cannot be inverted, the engine falls back to a coarser complete one and says so; strict mode throws instead. Differences that a setting hides (typography, page numbers, comments, …) are counted in a ledger, never dropped. Nothing needs a model: the deterministic engine keeps that promise on its own. It targets `net10.0`, uses no native code, and is built to be NativeAOT-, trimming- and WebAssembly-friendly. The same inputs and settings give byte-identical output. Nothing touches the network. ## Install ``` dotnet add package RecursiveAI.OneDiff # the text core dotnet add package RecursiveAI.OneDiff.Documents # document comparison, HTML and JSON dotnet add package RecursiveAI.OneDiff.Pdf # PDF in, PDF out dotnet add package RecursiveAI.OneDiff.Ocr # scans dotnet tool install --global RecursiveAI.OneDiff.Tool # the `onediff` command ``` | Package | Contents | |---|---| | `RecursiveAI.OneDiff` | The text core. Depends on nothing beyond the base class library. | | `RecursiveAI.OneDiff.Documents` | The comparison engine: compare model, alignment, grouping, typed changes, verification, combine and chains, the change report, JSON, HTML redlines. Depends on the core and `RecursiveAI.OneFont`. | | `RecursiveAI.OneDiff.Pdf` | Reads digital PDFs; marks changes on the original pages; writes a rendered redline PDF. Depends on `.Documents`, `RecursiveAI.OnePdf` and `RecursiveAI.OneRender.Pdf`. | | `RecursiveAI.OneDiff.Ocr` | Reads scanned PDFs and images through `RecursiveAI.OneOcr`, with each word's confidence; differences that may be recognition errors are flagged. | | `RecursiveAI.OneDiff.Tool` | The `onediff` tool: `diff`, `patch`, `merge`, `compare`, `combine`, `chain`. | Coming in later releases: Word documents in and tracked-changes DOCX out (`RecursiveAI.OneDiff.Word`, in this repository and not in the first release), spreadsheets, slides, and the OneNLP value recognizer (`RecursiveAI.OneDiff.Nlp`). Until then a Word file is refused with a message that says support is coming in a 1.x release and suggests comparing PDF exports. ## The happy path Each line below is a runnable sample in [samples/OneDiff.Samples](https://github.com/recursive-ai-oss/onediff-csharp/blob/6443915c4136cc8c5a49fa2543322011665f309e/samples/OneDiff.Samples/Samples.cs). ```csharp using OneDiff; using OneDiff.Documents; using OneDiff.Documents.Serialization; using OneDiff.Merge; using OneDiff.Pdf; // Text: the words that changed, a unified patch, a three-way merge. string marked = Diff.ToMarkdownInline(Diff.Words(oldText, newText)); string patch = Diff.Unified(oldText, newText); TextMerge both = Diff.Merge(baseText, ours, theirs); // both.Text, both.HasConflicts // PDF: a redline on the new document's own pages (highlights, with the old text in the pop-ups). PdfRedline.Register(); // once at startup Redline.Create("v1.pdf", "v2.pdf", "markup.pdf"); // PDF: a rendered, tagged redline PDF; or an HTML page with the two documents side by side. var cmp = Comparison.Of("v1.pdf", "v2.pdf"); cmp.SaveAs("redline.pdf", PdfRedline.Rendered); cmp.SaveAs("side-by-side.html", RedlineForm.HtmlSideBySide); // The change list: walk what changed in substance, or save it as JSON and read it back. foreach (var c in cmp.Changes.Where(c => c.IsSubstantive)) Console.WriteLine($"{c.Kind}: {c.Old.Text} -> {c.New.Text}"); Redline.Create("v1.pdf", "v2.pdf", "changes.json"); Comparison again = ComparisonJson.Read(File.ReadAllBytes("changes.json")); ``` Inputs are recognized by their content, and the output form follows the extension of the output path. The same from a shell: `onediff compare v1.pdf v2.pdf -o markup.pdf`. A PDF that has no text layer is refused with a message that names `RecursiveAI.OneDiff.Ocr` (call `OcrRedline.Register()` to read scans); it is never compared as an empty document. For streams that refuse synchronous I/O, such as an ASP.NET Core request body, use `Redline.CreateAsync`, `Comparison.OfAsync`, `SaveAsync` and `ComparisonJson.ReadAsync`. ## The low level Every layer is public: the comparison is five stages over immutable values, and the alignment between the two documents is a value you can correct and run again. ```csharp using OneDiff.Documents; using OneDiff.Documents.Model; // CompareDocument using OneDiff.Documents.Pipeline; // Alignment CompareDocument a = CompareDocument.FromText(oldText, "old"); CompareDocument b = CompareDocument.FromText(newText, "new"); Alignment alignment = Stages.Align(a, b); alignment = alignment.Pin(oldNodeId, newNodeId); // a reviewer's correction Comparison result = Stages.Verify(Stages.Classify(Stages.Group(Stages.DiffPairs(alignment)))); ``` ## Documentation Start at [docs/index.md](https://github.com/recursive-ai-oss/onediff-csharp/blob/6443915c4136cc8c5a49fa2543322011665f309e/docs/index.md). The pages that matter first: - Samples, every redline look, and how to choose a form: [docs/samples/README.md](https://recursiveai.org/onediff/docs/samples/) - Coming from DiffPlex: [docs/migrating-from-diffplex.md](https://recursiveai.org/onediff/docs/migrating-from-diffplex/); from `WmlComparer` or Docxodus: [docs/migrating-from-wmlcomparer.md](https://recursiveai.org/onediff/docs/migrating-from-wmlcomparer/) - What a comparison is and how it is proved complete: [docs/compare-model.md](https://recursiveai.org/onediff/docs/compare-model/) - Measured results, including the rows that are pending or lose: [docs/results.md](https://recursiveai.org/onediff/docs/results/). Nothing beyond that page is claimed. In short: Myers' edit length equals GNU diff's minimum on 9,483 corpus pairs, three-way merge equals `git merge-file` byte for byte on 2,599 real triples in all three conflict styles, and in the speed rows against DiffPlex and diff-match-patch every area's geometric-mean time ratio is below 1.0 (the worst single row is 1.13×, patch make against diff-match-patch on x64). ## Known issues - In browser WebAssembly, reading scans with `RecursiveAI.OneDiff.Ocr` fails with a `TypeLoadException` from the image library it uses (OneImage). Scans compare normally on every native platform; text, document and digital-PDF comparison work in the browser. ## Building ``` dotnet build -c Release dotnet test --solution OneDiff.slnx -c Release ``` Build output goes outside the checkout; set `ONEDIFF_ARTIFACTS` to choose where. See [docs/building.md](https://recursiveai.org/onediff/docs/building/). ## License Dual-licensed under either of [MIT](https://github.com/recursive-ai-oss/onediff-csharp/tree/6443915c4136cc8c5a49fa2543322011665f309e/LICENSE-MIT) or [Apache-2.0](https://github.com/recursive-ai-oss/onediff-csharp/tree/6443915c4136cc8c5a49fa2543322011665f309e/LICENSE-APACHE), at your option. © Recursive AI, LLC. # Samples The runnable code samples (one task per method) are in [samples/OneDiff.Samples](https://github.com/recursive-ai-oss/onediff-csharp/blob/6443915c4136cc8c5a49fa2543322011665f309e/samples/OneDiff.Samples/Samples.cs). This page covers the rendered files and the choice of form. Every file here is made from a short pair of documents written for the purpose (a list of clauses with an edit, an insertion, a deletion, a move and a formatting change), by the tests that check the writers. A change in the look shows up as a change to these files in review. To rewrite them after an intended change, run the tests with `ONEDIFF_UPDATE_SAMPLES=1`. | Files | What they show | |---|---| | `-inline.html`, `-side-by-side.html` | The HTML redline in each preset: `blackline`, `sidebyside`, `word`, `developer`. | | `-inline.pdf`, `-side-by-side.pdf` | The rendered redline PDF in the same presets and layouts: a new, tagged PDF laid out from the comparison (A4, landscape for side by side). | | `markup-on-pages.pdf` | Original-page markup: two short PDFs compared, and the changes marked as annotations on the new PDF's pages (the pages themselves are untouched). | ## Happy path, one or two lines each A PDF output needs `PdfRedline.Register()` once at startup. The output form follows the extension of the output path, and inputs are recognized by their content. ```csharp // Text core: the edits between two strings, as a unified diff. string patch = Diff.Unified(oldText, newText); // Redline as a self-contained HTML page (inline), and as the versioned change list (JSON). Redline.Create("v1.txt", "v2.txt", "redline.html"); Redline.Create("v1.txt", "v2.txt", "changes.json"); // Redline as a PDF. Any input kind renders to a blackline; a PDF as the new document is marked on its own pages. PdfRedline.Register(); Redline.Create("v1.txt", "v2.txt", "redline.pdf"); // rendered blackline Redline.Create("v1.pdf", "v2.pdf", "markup.pdf"); // highlights and notes on v2's pages // Choose the form or the look yourself. var cmp = Comparison.Of("v1.pdf", "v2.pdf"); cmp.SaveAs("redline.pdf", PdfRedline.Rendered); // rendered redline from PDF inputs cmp.SaveAs("side-by-side.pdf", PdfRedline.RenderedSideBySide, RedlineStyle.SideBySide); cmp.SaveAs("redline.html", RedlineForm.HtmlSideBySide); ``` # Migrating from DiffPlex This page maps the calls in DiffPlex's README, checked against the public members of DiffPlex 1.9.0, to the `RecursiveAI.OneDiff` calls that do the same job. It is a mapping only: OneDiff does not copy DiffPlex's types or shapes, and there is no compatibility layer. Where the README does not say exactly what a DiffPlex option does, the row says "closest" and you should check the result on your own data. Rows marked *not provided* have no equivalent in the text core. Everything below is in namespace `OneDiff` (and `OneDiff.Tokenization`, `OneDiff.Merge` where noted). All calls are static and take plain strings. ## The result: `DiffResult` → `TextDiff` and `EditScript` `Diff.Lines`, `Diff.Words`, `Diff.Characters`, `Diff.Graphemes` and `Diff.Text` return a `TextDiff`: both strings, their tokens and an `EditScript`. The script is a list of `Edit` runs (`Equal`, `Delete`, `Insert`) whose indices count tokens of the old and new text. `TextDiff.OldSpan(edit)` and `NewSpan(edit)` give a run's text back. Replacements are a `Delete` followed by an `Insert`. ## `Differ` (`IDiffer`) | DiffPlex | OneDiff | Notes | |---|---|---| | `CreateLineDiffs(old, new, ignoreWhitespace: false)` | `Diff.Lines(old, new)` | Line tokens keep their line terminator, so `a\r\n` and `a\n` are different lines. | | `CreateLineDiffs(old, new, ignoreWhitespace: true)` | `Diff.LinesUsing(old, new, TokenComparer.IgnoreWhiteSpaceChanges)` or `TokenComparer.IgnoreAllWhiteSpace` | Closest. `IgnoreWhiteSpaceChanges` treats runs of white space as equal (and ignores a different line ending); `IgnoreAllWhiteSpace` drops white space altogether. | | `CreateCharacterDiffs(old, new, ignoreWhitespace)` | `Diff.Characters(old, new)`; with a comparer, `Diff.TextUsing(old, new, comparer, TokenKind.Character)` | Characters are Unicode scalar values, so surrogate pairs stay whole. `Diff.Graphemes` compares user-perceived characters. | | `CreateWordDiffs(old, new, ignoreWhitespace, separators)` | `Diff.Words(old, new)`; with a comparer, `Diff.WordsUsing(old, new, comparer)` | Word boundaries are fixed rules (see `TokenKind.Word`), not a separator list, so token boundaries differ from DiffPlex's. For your own separators, cut the text yourself and use `Diff.Sequences` (below). | | `CreateCustomDiffs(old, new, ignoreWhiteSpace, chunker)` | `Diff.Sequences(oldChunks, newChunks, comparer: …)` | You split the text; the script indexes your chunks. | | `CreateDiffs(old, new, ignoreWhiteSpace, ignoreCase, chunker)` | `Diff.TextUsing(old, new, TokenComparer.Create(ignoreCase, whiteSpace), kind)` | `TokenComparer.IgnoreCase`, `.IgnoreWhiteSpaceChanges`, `.IgnoreAllWhiteSpace` and `.Create(ignoreCase, WhiteSpaceHandling)` cover the combinations. The edit script still indexes the original tokens. | Which algorithm runs is a parameter: `Diff.Lines(old, new, DiffAlgorithm.Patience)` (`Myers`, `Patience`, `Histogram`). To cap the work on untrusted text: `Diff.Lines(old, new, new DiffSettings { MaxWork = 100_000 }).Diagnostic.CostCapReached`. DiffPlex's `Differ` has no algorithm choice or cap to map these to. ### Chunkers (`IChunker`) | DiffPlex | OneDiff | |---|---| | `LineChunker`, `LineEndingsPreservingChunker` | `TokenKind.Line` (line tokens include their terminator) | | `CharacterChunker` | `TokenKind.Character` (or `TokenKind.Grapheme`) | | `WordChunker` | `TokenKind.Word` | | `DelimiterChunker`, `CustomFunctionChunker` | split yourself, then `Diff.Sequences` | `Tokenizer.Tokenize(text, kind)` returns the token ranges if you want to look at them. ## `InlineDiffBuilder` (the README's console sample) OneDiff's line type is `DiffRun`, not `DiffPiece`, so a file can import both `DiffPlex.DiffBuilder.Model` and `OneDiff` while you migrate it without an ambiguous reference. The README builds an inline model with `InlineDiffBuilder.Diff(before, after)` and prints each line of `diff.Lines` with a `+`, `-` or blank prefix according to `line.Type`. `TextDiff.Lines` is the same view: one entry per line with its kind, text (without the line break) and line number in each text. | DiffPlex | OneDiff | |---|---| | `DiffPaneModel.Lines` | `TextDiff.Lines` | | `DiffPiece.Type` (`Unchanged`, `Deleted`, `Inserted`) | `DiffRun.Kind` (`EditKind.Equal`, `Delete`, `Insert`) | | `DiffPiece.Text` | `DiffRun.Text` | | `DiffPiece.Position` | `DiffRun.OldLine` / `DiffRun.NewLine` (both, one or the other, from 1; null on the side that lacks the line) | | `DiffPaneModel.HasDifferences` | `!diff.Script.IsIdentical` | ```csharp var sb = new StringBuilder(); foreach (DiffRun line in Diff.Lines(before, after).Lines) { string prefix = line.Kind switch { EditKind.Insert => "+ ", EditKind.Delete => "- ", _ => " " }; sb.Append(prefix).Append(line.Text).Append('\n'); } ``` `TextDiff.Pieces` gives the same runs without cutting them at line breaks (for a word diff, runs of words). `diff.Script`, `OldSpan` and `NewSpan` are still there when you want token indices. For a finished rendering instead of a loop: `Diff.ToTerminal(diff)` (ANSI colours, changed words highlighted, context lines and hunks), `Diff.ToMarkdown(diff)` (a fenced `diff` block) and `Diff.ToMarkdownInline(Diff.Words(a, b))` (``/`` inline). ## `SideBySideDiffBuilder` (`ISideBySideDiffBuilder`) `Diff.SideBySide(old, new)` returns a `SideBySideDiff`: aligned `Rows`, each with a `Kind` (`Unchanged`, `Deleted`, `Inserted`, `Modified`) and a `Left` and `Right` cell. A cell has the line number, the text and `Parts`, which for a modified line are the word-level pieces (Equal and Delete on the left, Equal and Insert on the right), the way DiffPlex fills `SubPieces`. | DiffPlex | OneDiff | |---|---| | `SideBySideDiffModel.OldText.Lines`, `.NewText.Lines` (padded with `Imaginary` pieces) | `SideBySideDiff.Rows`, with `Left` or `Right` null where DiffPlex pads | | `DiffPiece.Type` `Modified` | `SideBySideRow.Kind == SideBySideKind.Modified` | | `DiffPiece.SubPieces` | `SideBySideCell.Parts` | ```csharp foreach (SideBySideRow row in Diff.SideBySide(before, after).Rows) { string left = row.Left?.Text ?? "", right = row.Right?.Text ?? ""; Console.WriteLine($"{row.Kind,-9} | {left,-30} | {right}"); } ``` With an already computed line diff (for example one made with a comparer): `Diff.ToSideBySide(diff)`. ## `UnidiffRenderer` | DiffPlex 1.9.0 | OneDiff | |---|---| | `GenerateUnidiff(old, new, oldFileName, newFileName, ignoreWhitespace, ignoreCase, contextLines)` | `Diff.Unified(old, new, oldFileName, newFileName, contextLines)` | | `Generate(old, new, oldFileName, newFileName, ignoreWhitespace, ignoreCase)` | `Diff.Unified(old, new, oldFileName, newFileName)` | | `Generate(diffResult, oldFileName, newFileName)` | `Diff.Unified(old, new, …)` (the diff is computed inside; to keep a `TextDiff`, render it with `Diff.ToMarkdown` or `TerminalRenderer`) | | options to ignore white space and case | not available on `Diff.Unified`; render a comparer-based diff with `TerminalRenderer` or `MarkdownRenderer` over `Diff.LinesUsing(…)` instead | `Diff.Unified` has no timestamps in the header lines, and also takes `algorithm:` or, to cap the work, `DiffSettings`. Unified text parses back with `UnifiedPatch.Parse` and applies with `Diff.ApplyUnified`. ## `ThreeWayDiffer` (`IThreeWayDiffer`) DiffPlex's `CreateMerge(baseText, oldText, newText, ignoreWhiteSpace, ignoreCase, chunker)` takes the two modified versions as "old text" and "new text"; they map to `ours` and `theirs`, in that order. Its result, `ThreeWayMergeResult`, has `IsSuccessful`, `MergedPieces`, `ConflictBlocks` and `DiffResult`. | DiffPlex 1.9.0 | OneDiff | |---|---| | `CreateMerge(baseText, oldText, newText, …)` | `Diff.Merge(baseText, ours: oldText, theirs: newText)` returns a `TextMerge` | | `result.IsSuccessful` | `!merge.HasConflicts` | | `result.ConflictBlocks` | `merge.Conflicts`: each with the base, ours and theirs lines and line numbers; `merge.ConflictCount` for the count | | `result.MergedPieces` | `merge.Text`, with conflicts marked in git's `merge`, `diff3` or `zdiff3` layout (`new MergeSettings { Style = … }`), or settled with `Resolution = ConflictResolution.Ours/Theirs/Union` | | `CreateDiffs(…)` (three-way diff) | `merge.Regions`: the stretches of the merge, each with its base, ours and theirs line range | | `ignoreWhiteSpace`, `ignoreCase`, `chunker` | `Diff.Merge(baseItems, ours, theirs, settings, comparer)` over items you cut yourself (for example lines) with the comparer you want; `ItemMerge.Items` holds the merged items | ```csharp TextMerge merge = Diff.Merge(baseText, ours: oldText, theirs: newText); if (merge.HasConflicts) foreach (MergeConflict c in merge.Conflicts) Console.WriteLine($"conflict at ours line {c.OursLine}: {c.OursText} / {c.TheirsText}"); else Console.WriteLine(merge.Text); ``` Changes on adjacent lines are a conflict, as in git; changes separated by an unchanged line merge cleanly. ## Beyond DiffPlex The text core also has semantic and efficiency cleanup (`Diff.Semantic`, `Diff.Efficient`), diff-match-patch patch text (`Diff.PatchText`, `Diff.ApplyPatchText`) and unified patch apply with fuzz (`Diff.ApplyUnified`), and the `onediff diff | patch | merge` command line over the same calls. Document comparison lives in `RecursiveAI.OneDiff.Documents` and the adapter packages. The snippets and mappings above that show behaviour are covered by tests in the repository (`MigrationPageTests`). # Migrating from WmlComparer and Docxodus This page maps the calls in the README of Docxodus 15.0.0 (the NuGet package page; it is the maintained descendant of Open-Xml-PowerTools' `WmlComparer`, which that README says was removed in v11.0.0) to the OneDiff calls that do the same job. Names on the Docxodus side are only the ones that README shows: `DocxCompare.Compare`, `WmlDocument`, `SaveAs`, `DocxDiff` with `GetRevisions()`, `GetEditScriptJson()`, `GetSemanticChanges()` and `GetSemanticChangesJson()`, the `PreAcceptInputRevisions` and `PreserveInputRevisions` options, N-way consolidate, `RevisionProcessor` and the `Redline` command-line tool. Members of the older `WmlComparer` settings and revision types are not mapped, because the current documentation does not describe them. It is a mapping only: OneDiff does not copy their types or shapes, and there is no compatibility layer. Where the documentation does not say what an option does exactly, the row says "closest" and you should check the result on your own documents. Rows marked *not provided* have no equivalent. ## What works at 1.0 Reading and writing Word files is in `RecursiveAI.OneDiff.Word`, which ships after 1.0 (it builds from this repository today). The 1.0 packages are `RecursiveAI.OneDiff`, `.Documents`, `.Pdf`, `.Ocr` and the `onediff` tool. | Task | At 1.0, without `.Word` | With `.Word` | |---|---|---| | Compare two `.docx` files | **Not possible.** The input is recognized by its first bytes and the call throws `NotSupportedException`: "No document source is registered for this input, which looks like an Office Open XML or other zip-based document. Support for Office documents (DOCX, XLSX and PPTX) is coming in a 1.x release. Meanwhile, export the documents to PDF and compare the PDFs (with RecursiveAI.OneDiff.Pdf)." | `WordRedline.Register()` once, then `Comparison.Of("v1.docx", "v2.docx")` | | Write a `.docx` with tracked changes | Not possible: asking for a `.docx` output throws `NotSupportedException`: "No output form is registered for '.docx'. Support for DOCX output is coming in a 1.x release. Meanwhile, export the documents to PDF and compare the PDFs (with RecursiveAI.OneDiff.Pdf)." | `comparison.SaveAs("redline.docx")` | | Compare `.docx` against a PDF, write HTML, JSON or a PDF redline | Not possible (the `.docx` side cannot be read). | Works: `.Word` and `.Pdf` registered together. | | Compare PDFs, text or scans; HTML, JSON, rendered-redline and on-page-markup outputs | Works. | Works. | So a team that needs Word-in, Word-out redlines today uses the `.Word` package from this repository; nothing in the 1.0 set reads a `.docx`. ## The one-liner and the object model `DocxCompare.Compare(a, b)` returns a redlined document. The OneDiff equivalents split the same job in two, so that you can keep and inspect the comparison before choosing an output. | WmlComparer / Docxodus | OneDiff | Notes | |---|---|---| | `new WmlDocument("v1.docx")` | A path, or `DocumentInput.FromBytes(bytes, name)` / `FromStream` / `FromString` | Inputs are sniffed by content, never by extension. | | `var redline = DocxCompare.Compare(new WmlDocument("v1.docx"), new WmlDocument("v2.docx")); redline.SaveAs("redline.docx");` | `Redline.Create("v1.docx", "v2.docx", "redline.docx")` | The output form follows the extension of the output path. Needs `WordRedline.Register()` once. | | the same, keeping the result | `var cmp = Comparison.Of("v1.docx", "v2.docx"); cmp.SaveAs("redline.docx");` | `Comparison` is the verified change list; any number of outputs can be written from it. `cmp.ToBytes(form)` gives bytes instead of a file. | | `DocxCompare.Compare` compares the *accepted* view of each input (`DocxDiff` alone compares the raw markup unless `PreAcceptInputRevisions` or `PreserveInputRevisions` is set) | The default: `WordSourceOptions.Revisions = WordRevisionMode.Accepted` | `WordRevisionMode.AsOriginal` reads the documents as they were before their tracked changes; `Refuse` throws if a document holds any. Register a configured source with `WordRedline.Register(new WordSourceOptions { Revisions = … })`. | | the redlined document opens in Word with real `w:ins` / `w:del` / moves | `DocxRedline`, built on the new document by default (`BuildOn = WordBuildOn.Old` builds on the old one) | Accept-all gives the new document and reject-all the old, over the compare model; what a revision cannot express comes back in `DocxRedlineResult.NotCarried` from `new DocxRedline().Create(cmp, stream)`. | ## Settings The Docxodus README documents no comparison settings by name beyond the two revision options above, so this table maps OneDiff's own settings to the job, not to a member. | What you want | OneDiff | Notes | |---|---|---| | the revision author | `cmp.SaveAs("redline.docx", new DocxRedline { Author = "Redline" })` | Default author `OneDiff`. | | a revision date | `new DocxRedline { Date = someDate }` | No date is written unless you set one, so equal inputs give byte-identical files. | | how finely blocks are matched | `CompareSettings.MatchThreshold` (how similar two paragraphs must be to count as the same one, default 0.5), `SplitMergeThreshold` and `MoveThreshold` | These govern how blocks are matched, not how finely words are compared. | | what to show and hide | `CompareSettings.Legal` (default), `Strict`, `TextOnly`, `Hidden` categories, `CollapseCascades`, `TypedChanges`, `DetectMoves`, `CostCap`, `ThrowOnFailure` | See [compare-model.md](https://recursiveai.org/onediff/docs/compare-model/) and [semantics.md](https://recursiveai.org/onediff/docs/semantics/). `Strict` ignores nothing and throws instead of falling back. | ## Reading the revisions | Docxodus (`DocxDiff`) | OneDiff | Notes | |---|---|---| | `GetRevisions()`, typed revisions | `cmp.Changes`, an `ImmutableArray` | OneDiff reads the changes from the comparison, not back out of the output file. | | an inserted, deleted or moved revision | `change.Kind`: `Insert`, `Delete`, `Replace`, `Move`, `Split`, `Merge`, `Format`, `Attribute` | A replacement is one change with both sides, not a delete plus an insert. | | the revision's text | `change.Old.Text` and `change.New.Text` | Every change carries both sides and both anchors. | | the stable anchors on a revision | `change.Old.Anchor` and `change.New.Anchor` | Anchors name the source (a part and path, a PDF quad set, an OCR word). | | a revision's author | *Not applicable* to a single comparison. `Redline.Combine` / `Combination` attribute each change to the reviewer's copy it came from. | | | `GetEditScriptJson()` | `Redline.Create(a, b, "changes.json")` or `ComparisonJson.Write(cmp)`; `ComparisonJson.Read(bytes)` reads it back and re-checks it | A versioned schema, [schema/comparison-v1.json](https://github.com/recursive-ai-oss/onediff-csharp/blob/6443915c4136cc8c5a49fa2543322011665f309e/docs/schema/comparison-v1.json). | | `GetSemanticChanges()` / `GetSemanticChangesJson()` | `change.Type`, `change.OldValue` / `NewValue`, `change.IsSubstantive`, and `cmp.Ledger` | Typed values (`30 d → 60 d`), the substantive filter and the ledger of differences hidden by settings. | | `RevisionProcessor` (accept and reject revisions) | *Not provided.* OneDiff writes redlines; accepting and rejecting is OneDocument's job (`RevisionAuthor`), which OneDiff uses to verify its own output. | | ```csharp using OneDiff.Documents; using OneDiff.Word; WordRedline.Register(); // once at startup var cmp = Comparison.Of("v1.docx", "v2.docx"); cmp.SaveAs("redline.docx", new DocxRedline { Author = "Redline" }); foreach (var change in cmp.Changes.Where(c => c.IsSubstantive)) Console.WriteLine($"{change.Kind}: {change.Old.Text} -> {change.New.Text}"); ``` ## Consolidating reviewers' copies The Docxodus README describes an N-way consolidate against one base. The OneDiff call is `Redline.Combine("original.docx", ["alice.docx", "bob.docx"], "combined.html")` (or `Combination.Of`), which attributes each change to its reviewer and flags conflicting edits without picking a winner. The output today is the change report (`.html`) or the JSON form; a DOCX with one revision author per reviewer comes with a later `.Word` release. See [combine.md](https://recursiveai.org/onediff/docs/combine/). ## Not covered here Rendering, editing sessions, markdown projection, content-control helpers and the other parts of the Docxodus toolkit are not comparison, and OneDiff has no counterpart. The npm, Python and CLI surfaces of those packages have no OneDiff counterpart either; the OneDiff command line is `onediff compare v1.pdf v2.pdf -o markup.pdf` for the 1.0 formats. The snippets and mappings above that show behaviour are covered by tests in the repository (`MigrationPageTests` in the `.Word` tests, and the unsupported-input message in the samples tests). # Diff algorithms Behaviour of `MyersDiff`, `PatienceDiff` and `HistogramDiff`. All three take two `ReadOnlySpan` of item ids (equal ids are equal items) and return a canonical `EditScript`: equal runs, and between them a deletion followed by an insertion. ## Myers Myers 1986, "An O(ND) Difference Algorithm and Its Variations": the greedy furthest-reaching search (section 3) with the linear-space middle-snake refinement (section 4b). The script has the fewest deleted plus inserted items possible (`D`), unless the work cap below is reached. - A common prefix and suffix are removed first; they belong to some minimal script. - Items that occur on one side only are set aside before the search (they cannot be in a longest common subsequence) and the search runs on what is left. Minimality is unaffected. This is skipped when the ids are negative or too sparse to count in an array. - The search finds the middle snake of the (sub)problem, then solves the two halves, depth `O(log D)`. Memory is linear in the input. - Tie-breaking: among minimal scripts, the one whose forward and reverse searches meet first. It depends only on the input, so repeated runs give the same script. Where several minimal scripts exist this one is not chosen for readability; cleanup does that. ## Work cap Every search counts work in edit-graph steps: one per diagonal visited plus one per item matched while following a snake (Myers), and for the anchor-based algorithms one per item scanned plus one per item compared while growing a match. - The default cap is 2^31 steps, far above what ordinary inputs need (a pair differing by D edits costs on the order of D² steps). - `Compute(old, new, maxWork, out diagnostic)` takes an explicit cap. `long.MaxValue` means unbounded. `DiffDiagnostic.CostCapReached` reports whether it was hit and `Work` how much was spent. - When the cap is reached, parts already solved stay as they are and every part not yet searched becomes one deletion followed by one insertion. The script is valid but, for Myers, may be longer than the minimum. Patience and histogram are not minimal anyway. - Cancellation is checked once per edit count `d` in Myers and once per segment in the anchor-based algorithms. ## Patience From the published write-ups of Bram Cohen's patience diff. In each pair of ranges (after trimming a common prefix and suffix): 1. Items that occur exactly once in each range are candidates, taken in old order. 2. Their longest subsequence that is increasing in both ranges (patience sorting over the new positions) becomes the anchors. 3. The ranges between anchors are processed the same way. 4. A range with no candidate is finished with Myers. ## Histogram From the published descriptions of histogram diff. In each pair of ranges (after trimming): 1. Count how often each item occurs in the old range. 2. For each item of the new range that occurs in the old range at most 64 times, take each old occurrence as a starting point, grow the match in both directions while items are equal, and score it by its rarest item (lowest old-range count). A lower score wins; on equal scores the longer match; then the earlier one (earlier in the new range, then in the old). 3. After examining a starting point, scanning resumes after the furthest end of the matches grown from it. 4. The best match is the anchor; the ranges before and after it are processed the same way. 5. If the ranges share items but every shared item occurs more than 64 times in the old range, the range is finished with Myers. If they share nothing it is one deletion plus one insertion. The limit of 64 is a fixed constant of this implementation. ## Lines and the final newline Line tokens keep their terminator, so a last line with and without a newline are different items. Tokenizers decide where a line ends; the corpus comparison against GNU `diff` used `\n` only as the line end, as GNU does. ## Settings on the one-liners `DiffSettings` (`Algorithm`, `MaxWork`) is taken by `Diff.Text`, `Lines`, `Words`, `Characters`, `Graphemes`, `TextUsing`, `LinesUsing`, `WordsUsing`, `Semantic`, `Efficient` (the cap applies to the diff that is cleaned up; the cleanup passes run once over its edits), `Unified` and `Sequences` (which also reports the `DiffDiagnostic`). `TextDiff.Diagnostic` says whether the cap ended the search early (`CostCapReached`) and the work spent (the items of the shared head and tail count too; a diff with nothing to compare, such as an empty side, reports zero); it is all zero for a diff built with `TextDiff.Create`, and cleanup keeps it. A capped line diff is `Diff.Lines(a, b, new DiffSettings { MaxWork = 100_000 })`. Merges take the same two knobs in `MergeSettings`. # Patches How OneDiff reads, writes and applies unified diffs and diff-match-patch patch text. The code is in `src/OneDiff/Patching`; every rule below is checked by the gates in [measurement.md](https://recursiveai.org/onediff/docs/measurement/). ## Unified diff The format is the GNU unified format (GNU diffutils manual, "Unified Format" and "Detailed Unified"). **Lines.** Text is cut at `\n` only. Each line keeps its terminator, so `\r\n` files round-trip byte for byte and a lone `\r` is ordinary content. A last line without a terminator differs from the same line with one, which is what makes `\ No newline at end of file` come out right. **Making.** Changes whose gap of unchanged lines is at most twice the context length share a hunk. The default context is three lines; zero is allowed. A side with no lines in a hunk names the line before it (`-0,0`, `+5,0`). A count of one is left out (`-3`). Names on the `---`/`+++` lines are the caller's; timestamps appear only when given. Identical inputs give the empty string. A name containing a control character, quote or backslash is written in git's quoted form. **Parsing.** - Text before the first `---`/`+++` pair and between files that is not a header is skipped, as the patch manual says. Git's extended header lines (`diff --git`, `index`, mode, rename and similarity lines) directly before a file are kept verbatim on that file. Binary and rename-only sections are not supported and are skipped. - A hunk header is `@@ -a[,b] +c[,d] @@` followed by an optional section heading. Numbers above about a billion are rejected. A hunk takes exactly the lines its counts announce; fewer is a truncated patch, more is an unexpected line. An empty line inside a hunk is an empty context line. `\ No newline at end of file` removes the terminator from the line before it. - Trailing carriage returns on header lines are dropped. A `@@` line outside a file's hunks, or a `---` line without `+++`, is an error. - Input over 2^28 characters is refused. Memory use is proportional to the input; no allocation depends on a number in the input. **Applying.** Hunks are matched against the text line by line, exactly. For each hunk: 1. It is looked for where its header says, shifted by the offset the previous hunk was found at, then at growing distance, later lines before earlier ones at the same distance. 2. If it does not match, context lines are ignored. *Fuzz n* leaves at most (longest context − n) lines at each end, so a hunk whose contexts differ (because it sits at the edge of the file) loses lines from its longer side first. At most two levels by default. 3. A hunk with less leading than trailing context (after fuzz) must match at the top of the file; the mirror case at the end. 4. A hunk may not match where its ignored leading context would fall before the first line. Ignored trailing context may run past the last line. 5. A hunk never matches before the lines earlier hunks used. 6. A hunk that matches nowhere is rejected, with the reason `ContextMismatch`, or `AlreadyApplied` when its new lines are found where its old lines are not. Applying never throws for a conflict; the rejected hunks are returned as a patch. Rules 2–4 are not in the patch manual. They were settled by comparing results with GNU patch 2.8 (run as `patch -f`, which never asks and never assumes a reversed patch) on thousands of generated drifted files (`tests/OneDiff.Tests/Patches/GnuPatchAgreementTests.cs`). **Reverse.** `Reverse` swaps the sides of every hunk; applying a reversed patch to the new text gives the old. ## diff-match-patch patch text The format and the matching are those of diff-match-patch's published wiki ("Unidiff" and "API") and of Wu and Manber, "Fast text searching allowing errors" (1992). **Format.** Each patch is `@@ -s1[,l1] +s2[,l2] @@` and one line per diff: a space, `-` or `+` followed by the text, URI-encoded as UTF-8 with the space left as it is. A length of one is left out; with a length of zero the start is written as it is, otherwise it is one-based. The context is rolling: the starts of a patch are those of the text after the earlier patches were applied. A lone surrogate is written as the three bytes its code unit would take, and read back the same way. Reading takes the starts from the headers and recomputes the lengths. **Offsets.** Starts count UTF-16 code units by default (the JavaScript, Java and C# ports). The Python package counts code points; `DmpOffsetUnit.CodePoint` makes `Make` and `Apply` use them, which is needed to exchange patches with it for text outside the Basic Multilingual Plane. **Making.** The character diff trims common ends, compares long texts line by line first, and never splits a surrogate pair. No readability cleanup is applied. Context grows by the margin (default 4) until the patch's old text is unique in the text, up to the match limit, plus one more margin. **Applying.** The text is padded at both ends, patches longer than `MatchMaxBits` (default 32) are split, and each patch is placed by `Match` near where it expects to be, shifted by the drift of the patches before it. A patch whose old text is longer than the limit is located by its first and last stretch. When the found text differs from the expected text, the patch's edits are mapped through a diff between the two; a long deletion over text that differs by more than `PatchDeleteThreshold` is refused. Per patch the result says applied, not found, or too different. Cost is the text length times the number of patches. **Match.** An exact hit at the expected place wins; otherwise Bitap scores each candidate as errors / pattern length + distance / `MatchDistance` and takes the best one under `MatchThreshold` (defaults 0.5, 1000). ## Limits and errors Both parsers return a typed error (`PatchParseError`: kind, line, offset) through `TryParse`/`TryFromText`, or throw `PatchFormatException` from `Parse`/`FromText`. Nothing else is thrown for bad patch text. The parsers and appliers are fuzzed (`fuzz/OneDiff.Fuzz`). ## Options on the one-liners `Diff.ApplyUnified(text, patch, options)` takes `UnifiedApplyOptions` (fuzz, offset limit, reverse). `Diff.ApplyPatchText(text, patchText, patcher, cancellationToken)` and `Diff.PatchText(old, new, patcher, cancellationToken)` take the `DmpPatcher` whose thresholds and margin apply; without one the defaults are used. Where a hunk or patch was not placed, `HunkResult.Line` and `DmpPatchResult.Location` are `null`. `DmpPatcher.TryMatch` returns false for no match and for a pattern longer than `MatchMaxBits`. # Three-way merge `ThreeWayMerge` (and `Diff.Merge`) merge a common ancestor (base) with two descendants (ours, theirs) over any sequence; the text form works on lines. The structure follows Khanna, Kuber and Pierce, "A Formal Investigation of Diff3" (FSTTCS 2007); the conflict layouts follow git's documentation for `git merge-file` and `git merge` (conflict styles). Where those leave behaviour open, it was settled by running `git merge-file` 2.51.2 on crafted inputs and the 2,599 triples of the merge corpus, and is marked *observed* below. ## Lines A line ends at `\n` and keeps its ending. A lone `\r` is not a line break. Two lines are equal only if their endings are too (`a` and `a\n` differ, `a\r\n` and `a\n` differ). The last line may have no ending. ## Aligning Each side is aligned with the base, and the alignment is turned into hunks: maximal runs of changed items, each a range of the base and the range of the side that replaces it. With the default algorithm (Myers) the merge uses its own alignment, not the shortest-script one of `Diff`, because several equally short scripts exist and a merge is only as good as the extents of its hunks. Patience and histogram use the library's own alignments. The rules below were each found by comparing with the output of `git diff` and `git merge-file` 2.51.2 on crafted inputs (every pair over three lines up to length 4, 20,000 random pairs, 60,000 longer pairs with a frequent line) and then confirmed on the 2,599 merge triples. All are *observed in output of git 2.51.2*. 1. **Trimming.** The common start and end of the two sequences are matched first and take no part in anything below. 2. **Items that cannot match.** An item with no copy in the other sequence is changed. The counts are over the whole other sequence, before trimming. 3. **Items with too many copies.** An item is *frequent* when the other sequence holds at least `limit(N)` copies of it, N being the length of the sequence the item is in. There is no limit under 8 items; the limit is 4 for N from 8 to 15, 8 for 16 to 63, 16 for 64 to 255, 32 for 256 to 1023, 64 from 1024 (a power of two near the square root of N). 4. **Which frequent items stay.** Inside the trimmed range, look left and right from a frequent item over items that have no copy (counted as D, left and right) and over other frequent items (counted in P, which starts at 2), stopping at the first ordinary item. The item stays in the search when either D is zero, or when D left plus D right is at most three times P; otherwise it is changed. The looks are static: another frequent item being dropped does not change them. 5. **The search.** Myers 1986 §4b on what is left (forward and reverse searches meeting at a middle snake, recursing on both halves), except that the forward diagonals of each step are visited from k = d down to k = -d; the reverse ones go up. For base `aab` against `ba` this matches the `b`, where visiting in the paper's order matches an `a`. 6. **Sliding.** A run of changed items that could sit in several places is placed by sliding it up as far as it goes (joining runs it meets) and then down as far as it goes, noting each place where the other sequence has a run lined up with it; if sliding down joins the next run, the joined run starts again (up, then down with fresh memory); it settles at the last lined-up place, or at the lowest place if there is none. The first sequence is settled against the other one as searched; the second against the first as settled. The result is not always the shortest script. Items skipped by rules 2 to 4 never match, so the hunks can be longer than the minimum. ## Walking the hunks Both hunk lists are walked together in base order. - A hunk that has no counterpart touching it is **one-sided**: ours-only or theirs-only; the merge takes that side. - An ours hunk and a theirs hunk with the same base range and the same new items are one **false conflict** (`BothSame`): taken once. - Anything else is a **conflict**. Observed: hunks that merely touch (one ends where the other starts, or both insert at one point) conflict. A conflict covers every hunk of either side that overlaps or touches it, repeatedly, so its base range is the union and its ours and theirs ranges are what each side holds there. - Observed: a conflict whose two sides end up equal after different splits into hunks is still a conflict in the diff3 and zdiff3 styles. In the merge style, narrowing (below) finds nothing left to mark and it disappears. ## Conflict styles - **merge.** Each conflict is narrowed by aligning its ours side with its theirs side: shared items stay out and each differing stretch is a conflict of its own. Then neighbouring conflicts are joined (observed): two conflicts with only shared, unchanged items between them become one when the gap is at most three items, or when no item of the gap holds an ASCII letter or digit. A gap that holds a one-sided change does not qualify; a false conflict settled earlier counts as shared text. Layout: ours section, `=======`, theirs section. - **diff3.** No narrowing and no joining. Layout: ours, `|||||||` with the base, `=======`, theirs. - **zdiff3.** diff3, but items that begin both sides identically move out before the conflict and items that end both sides identically move out after it (observed: the start is taken first, the end from what is left). The base section stays whole. A conflict can end up with empty ours and theirs sections. Markers are `<<<<<<<`, `|||||||`, `=======`, `>>>>>>>` (marker size adjustable) followed by a space and the label when there is one. A section whose last line has no ending gets one inside a conflict, so the next marker starts a line. ## Line endings in markers Observed: marker lines (and the ending added to an unterminated last line) use CR LF exactly when, for the conflict: the base line before it (or the first base line when it starts the file) ends in CR LF, and so does the corresponding line of ours and of theirs where there is one. An empty base never asks for CR LF. For a conflict that was narrowed or joined, the base line is the one before where the whole conflict began. ## Resolutions `Ours`, `Theirs` and `Union` settle conflicts after narrowing, joining and trimming (so `Union` on a narrowed conflict gives each narrowed piece's two sides, not the whole conflict twice). `Union` writes ours first and adds a line ending after an unterminated last line of ours. The conflict count is then zero. ## Structured result `Regions` tile the base, ours and theirs: each region holds the three ranges. Narrowed or trimmed conflicts share the conflict's base range: the first conflicting piece carries all of it, and shared items around the pieces are `BothSame` regions with an empty base range. ## Agreement with git On the 2,599 real merge triples of the corpus, the text produced equals the output of `git merge-file` 2.51.2 byte for byte in the merge, diff3 and zdiff3 styles, with git choosing its own alignment. The base-to-side alignments themselves still differ from `git diff` on a few corpus pairs (26 of 5,198); none of them changes a merge. Very large inputs may differ from git where git gives up searching for the shortest script, which is not modelled here; the cost cap below is OneDiff's own. ## Cost cap The alignments of one merge share a budget, `MergeSettings.MaxWork` (2^31 by default, the default of the diff algorithms; zero or more). Work is counted as in the Myers search: one unit per diagonal visited, per item compared along a diagonal, and per item slid. Items that cannot match and the frequent-item rule (steps 2 to 4 above) are linear and not counted. When the budget runs out: - the search stops, and every stretch it had not solved is reported as changed on both sides (the same coarsening Myers uses: unsearched parts become delete and insert); - sliding (step 6) stops where it is; each slide step keeps the alignment valid; - the merge carries on with the hunks it has. Hunks that are longer than needed only make conflicts larger, or turn a clean merge into a conflict. A change of either side is never dropped: it is applied or inside a conflict, and the regions still tile all three inputs; - `TextMerge.Diagnostic` and `ItemMerge.Diagnostic` have `CostCapReached` set and `Work` holds the work spent, a little over the cap at most (the cap is checked once per search round, so the overshoot is bounded by the size of the inputs). A merge that stays within the budget is byte-for-byte what it would be without a cap. With the default cap this never happens on the merge corpus. The patience and histogram alignments receive what is left of the budget and report the same way. ## Conflicts and merged items `TextMerge.Conflicts` lists the conflict regions in order as `MergeConflict`s: index, region, the 1-based line each version starts on, and the three versions' lines (line endings included). It lists the places the sides disagreed whether or not `MergeSettings.Resolution` settled them in `Text`, so with a resolution `ConflictCount` is zero while `Conflicts` still shows what was settled. `Diff.Merge` returns an `ItemMerge`: `Items` is the merged sequence with conflicts settled as `Resolution` says (null when the resolution is `None` and there are conflicts, because there is no single sequence then), and `Resolve(how)` settles them another way (Ours, Theirs, or Union with ours first). # Text output: terminal and Markdown Renderers over a line-token `TextDiff`. Output is a pure function of the diff and the renderer's settings: the same input gives the same bytes on every platform. Every output line ends in `\n`, whatever the input's line endings. ## Hunks (both formats) - Lines are the line tokens of the diff. A line's terminator (`\n`, `\r\n` or a lone `\r`) is not shown. - A change is surrounded by `ContextLines` equal lines (default 3). Two changes whose gap is at most twice that share one hunk, so no line appears twice. With 0 context every change is its own hunk. - A hunk starts with `@@ -a,b +c,d @@`: one-based first line and line count on each side. A count of 1 is written without `,1`. An empty range names the line before it (`-0,0` at the start of the text). - Lines are prefixed `' '` (context), `-` (only in the old text) and `+` (only in the new text). Deletions of a changed block come before its insertions. - A shown line that had no terminator is followed by `\ No newline at end of file`. - When `OldName` or `NewName` is set, `--- old` and `+++ new` precede the first hunk. - An identical pair renders as the empty string. ## Terminal - Colour (default on): hunk headers cyan (`ESC[36m`), file headers bold, deletions red (`ESC[31m`), insertions green (`ESC[32m`), the no-newline note dim; each styled line ends with `ESC[0m`. Colour off writes the same text without any escape sequence. - Word level (colour on only): in a changed block that has both deleted and inserted lines, the block's old and new lines are diffed word by word and the changed words are wrapped in reverse video (`ESC[7m` … `ESC[27m`). Skipped when either side of the block exceeds 8192 chars, or when under 30% of the longer side (counting non-white-space chars) is shared — those lines are rewrites and marking everything is noise. - Control characters in the text (below U+0020 except tab, U+007F, U+0080–U+009F) are replaced by a visible picture (U+2400 block, U+2421, U+FFFD) so compared text cannot drive the terminal. ## Markdown - Line diff: the hunks above in a fenced block opened with `diff`. The fence is three backticks, or one more than the longest backtick run in the content. - Inline diff (any token kind): the new text, each deletion wrapped in `` and each insertion in ``. A run that spans line breaks is wrapped per line, with the breaks outside the tags. `&` and `<` are escaped as `&` and `<`; other Markdown is left as written. ``/`` is used rather than `~~`/`**` because those collide with markup already in the text and cannot be escaped reliably. ## Reading views Three views read a `TextDiff` without token arithmetic. - `TextDiff.Pieces`: one `DiffRun(Kind, OldLine, NewLine, Text)` per edit run, with the run's text as it stands (line breaks included) and the 1-based line each side starts on (`null` on the side that lacks the run). Works for every token kind. - `TextDiff.Lines`: the pieces cut at line breaks (`\n`, `\r\n`, a lone `\r`), the breaks dropped. For a line diff that is one entry per line; for a word or character diff a changed line shows as several entries with the same line number. A side's line number advances when a piece starts a new line on that side, so an unterminated last line is one line. - `Diff.SideBySide(old, new)` / `SideBySideDiff.From(lineDiff)`: aligned rows. A run of deleted lines followed by a run of inserted lines is paired line by line into `Modified` rows (the longer run's remainder is `Deleted` or `Inserted` rows with the other cell null). A modified row's cells hold word-level `Parts` from `Diff.Words` plus semantic cleanup of the two lines: Equal and Delete on the left, Equal and Insert on the right. Needs a diff made with line tokens. # The compare model, the stages and the completeness guarantee This note describes the contracts of `RecursiveAI.OneDiff.Documents`: what a compared document is, how a comparison is built from five stages, and how every comparison is proved complete. Sources, alignment, in-pair diff, grouping, writers and models plug in through the types named here. ## 1. The guarantee For every comparison: 1. **Every difference is accounted for.** Each difference between the two compare models is either a *change* or an entry in the *ledger* (hidden by a setting: category, count, locations). Content outside the model is reported "differs, not itemized" from a content hash. 2. **Invertible.** Old with the changes applied is new; new with the changes removed is old. This is checked by replaying the changes' edits over the compared content in both directions, not by trusting the stage that made them. 3. **Both sides, both anchors.** Every change carries the old and new text it concerns and an anchor for each side that resolves back to the source. 4. **Grouping is display.** Changes are atomic data. A `ChangeGroup` lists change ids and carries no data, so removing or reshaping groups never loses or alters a change. `Stages.Verify` checks all four on every comparison. On failure it falls back, step by step, to a coarser grouping that is still complete, and records why in the diagnostics; with `CompareSettings.ThrowOnFailure` it throws `ComparisonVerificationException` instead. Without any model the deterministic stages keep every promise; a model may reorder, group and score but cannot hide a change, because whatever it produces is verified like everything else. ## 2. The compare model A `CompareDocument` is an immutable forest of `Node`s in document (pre-order) order. `Nodes[i].Id == i`, and a parent always has a smaller id than its children, so a node's id is its position. A document also has a display `Name`, the `SourceHash` of the input bytes, a `SourceFormat`, and `Opaque` parts. A `Node` has: | Member | Meaning | |---|---| | `Kind` | `Section`, `Paragraph`, `ListItem`, `Table`, `TableRow`, `TableCell`, `Footnote`, `Endnote`, `Header`, `Footer`, `Comment`, `Field`, `Image`, `TextBox`, `ContentControl`, `Property` | | `Text` | normalized text the node's own content is compared on (empty for pure containers); never contains U+001E | | `Runs` | character formatting over `Text`: sorted, non-overlapping, adjacent equal runs merged, never an empty format | | `Attributes` | block-level attributes as a sorted string map (paragraph style, list label and level, hashes, field code, property name) | | `Anchor`, `Segments` | where the node came from; segments anchor pieces of the text (a line, a word) | | `Folds` | what normalization changed (below) | Conventions for the flowing kind: - **List items** carry their resolved label in `list.label` (and `list.level`, `list.format`), so a renumbering is an attribute change, not a text change. - **Fields** are `Field` nodes under their paragraph: the code is `field.code`, the result is the node's text. A source that wants the field to appear in the paragraph's text puts U+FFFC there; the child node holds the content. - **Hyperlinks** are the `run.link` attribute of the linked text. - **Images** are `Image` nodes with `hash.content` and, when a source has one, `hash.perceptual`. - **Comments** and **document properties** are nodes (`Comment`, with block children; `Property`, named by `prop.name`, the value is the text). They can be hidden into the ledger. - Attribute keys have a prefix that says which hiding category they belong to: `run.` (character formatting), `para.`, `list.`, `section.`, `field.`. Other prefixes (`hash.`, `table.`, `prop.`, …) are content or structure and are never hidden. **Slides and grids** are not read by the 1.0 packages. The model is shaped for them (`DocumentKind`, `Anchor.ForCell`, attributes for geometry, R1C1 formulas and number formats) so that they can arrive as sources without a change to the stages. `Stages.Align` refuses to compare documents of different kinds. ### Anchors An `Anchor` is a source location the engine copies but never interprets. The anchor kinds are `Node` (an opaque adapter reference, for example a hub node), `PdfQuads` (glyph quadrilaterals with their pages), `OcrWords` (word quadrilaterals plus the lowest confidence), `Cell` (sheet and A1 address) and `TextSpan` (a character range of a named text part). `.Documents` references no source library; adapters choose the `Reference` string. Anchors are the contract between sources and writers: a source attaches them, the engine copies them onto changes, a writer resolves them. `Anchor.Union` combines neighbouring anchors (hull of text spans, concatenated quads with the lowest OCR confidence, otherwise the first). A change side's anchor is the union of the anchors of the node segments its range overlaps, or the node's anchor when it has no segments. Verification recomputes that and compares. ### Normalization and the ledger A normalizer (`ITextNormalizer`) looks at one position of raw text and may replace the characters there. `TextNormalization.Normalize` runs them and returns the normalized text plus a `NormalizationFold` per replacement: its ledger category, its place in the normalized text, and the original characters. A node keeps both, so `Node.RawText` always reconstructs the source text. The built-in `TypographicNormalizer` folds quotes, the dash family and runs of horizontal whitespace; PDF sources add `LineBreaks`, `Hyphenation` and `PageNumbers` folds in the same way. What the settings hide is a **projection** of the model: - A fold whose category is hidden stays folded; a fold whose category is *not* hidden is undone, so its difference shows as ordinary text. - Run and block attributes whose category is hidden are removed from the compared content. `Comment` and `Property` subtrees are hidden whole. - The ledger then lists every difference that exists in the full model but not in the compared projection, outside the regions a change already shows: each place where a hidden fold, hidden format run or hidden attribute differs between paired nodes, each hidden node added, removed or changed, and each opaque part that differs. Differences inside a region a change shows belong to that change. `LedgerBuilder` computes the ledger from the alignment, the edits and the settings only, so verification recomputes it and fails when an entry is missing. ## 3. The alignment An `Alignment` says which nodes correspond. It holds both documents, the settings, and `AlignmentLink`s: `Match` (one old node is one new node in the same place), `Move` (same node elsewhere), `Split` (one old, several new) and `Merge` (several old, one new). A node is in at most one link. Nodes in no link are deleted (old) or inserted (new). A link joins nodes of one `NodeKind`. Edit it with `Pin(old, new)`, `Force(oldNodes, newNodes, kind)` and `Unpin(linkId)`; each returns a corrected copy, normalized. Run `Stages.Align(old, new, seed: alignment)` to keep pinned links and re-pair the rest, or just run `DiffPairs` onwards on the copy. **Normalization of links** (`Alignment.Normalize`) decides which one-to-one links are `Move`. A link is a `Match` only if its old parent is paired with its new parent and it belongs to the longest run of siblings that kept their order; every other one-to-one link is a `Move`. Hidden-subtree links and splits and merges are exempt. Verification checks the same two rules on every `Match`, so a reorder can never hide: it has to be a visible `Move` change. ## 4. The stages All stages are pure functions of immutable values, thread-safe, and take a `CancellationToken`; the cost cap (`CompareSettings.CostCap`) travels with the settings in the alignment. Exceeding a cap never throws: the stage returns a coarser, still complete result and adds a `Warning` diagnostic. ```csharp Alignment Stages.Align(old, new, scorer?, settings?, seed?, ct) PairDiffs Stages.DiffPairs(alignment, ct) ChangeSet Stages.Group(pairDiffs, ct) ChangeSet Stages.Classify(changeSet, ct) Comparison Stages.Verify(changeSet, ct) Comparison.Of(old, new, settings?, scorer?, ct) // the five in a row ``` | Stage | What it does | Contract the next stage relies on | |---|---|---| | `Align` | The text core over subtree hashes of all nodes in document order, equal subtrees that moved, then text blocks: candidates from a shingle index, splits and merges from containment, ambiguous pairs judged by `IPairScorer` in batches, a global maximum-weight assignment; then containers by matched content and position; then `Normalize`. Details: [alignment.md](https://recursiveai.org/onediff/docs/alignment/) | links satisfy the rules above | | `DiffPairs` | Per link, a word-level diff over UAX #29 segments with semantic cleanup; format-only edits for equal text with different runs; attribute edits; the ledger. Details: [grouping.md](https://recursiveai.org/onediff/docs/grouping/) | for every non-hidden link, edits that replay old → new and back; `Links[i]` is indexed by link id | | `Group` | Rule-based change units (values, overlap, sentence and clause breaks, density), one `Change` per unit, whole inserted/deleted nodes, moves, splits, merges and attribute changes; display groups, with cascades and defined-term renames lifted when `CollapseCascades` is on. Details: [grouping.md](https://recursiveai.org/onediff/docs/grouping/), [semantics.md](https://recursiveai.org/onediff/docs/semantics/) | every inserted/deleted node and every link is explained; ids unique; groups partition the changes | | `Classify` | Change types and resolved values (when `TypedChanges` is on, through the value recognizer), `IsSubstantive`, OCR flags and the `Decision` on every change. Details: [semantics.md](https://recursiveai.org/onediff/docs/semantics/) | none beyond `Group` | | `Verify` | The checks in section 5, the fallback, and the `Comparison` | — | Seams: `IPairScorer` (`Score`, `ScoreBatch`) for alignment candidates, `ITextNormalizer` for sources, `IDocumentSource` for inputs. A scorer can only reweigh pairings; every pairing is verified afterwards. `IValueRecognizer` is the seam on `Classify` for typed values (`CompareSettings.ValueRecognizer`); `Change.Decision` records rule, model or user with a score. ### Change shapes - **Whole node** inserted or deleted: `LinkId = -1`, one side present, no edits. - **Region** (Insert, Delete, Replace, Format) inside a link: one or more `RegionEdit`s. Coordinates are in the *joined compared text* of the link's nodes (one node: its compared text; a split or merge: the nodes joined with U+001E). An insertion has a zero-length old range, a deletion a zero-length new range, a format change equal old and new text with different runs. `OldRuns`/`NewRuns` carry the formatting of the replaced and replacing characters relative to the edit. - **Move**: no edits; both sides cover the whole node. Edits to a moved node are separate changes with the same `LinkId`. - **Split/Merge**: one change for the link carrying all of its edits. - **Attribute**: `AttributeEdit`s per (old node, new node) pair; absent values are `null`. ## 5. Verification `Verifier` takes only an alignment, changes, groups and a ledger, and checks: | Code | Check | |---|---| | V100–V104 | change ids unique and non-negative; groups unique; groups list only known changes; every change in exactly one group; sides and edit lists present | | V105–V113 | a change attached to no link is a whole-node insert/delete of an unmatched node with no edits; link ids valid; the change names the link's nodes; sides have well-formed ranges, the text of that range, and an anchor equal to the one the model resolves; a change's sides contain all of its edits (they may show more, such as a whole number or sentence) | | V200 | every visible old node is in exactly one link or deleted exactly once; every visible new node likewise inserted | | V300–V304 | hidden links have no changes; a `Move`/`Split`/`Merge` link has exactly one change of that kind; the link's edits replay old → new **and** new → old over text and formatting; attribute edits replay both ways | | V400–V401 | every `Match` keeps its parent and its sibling order | | V500 | the ledger lists every hidden difference | **Fallback** (when not strict), each step checked again: **1** regroup with the plain fallback stages on the same alignment (drops a bad grouping or a bad in-pair diff); **2** replace every changed block as a whole (drops a bad word diff); **3** nothing aligned, so every block is deleted and inserted (drops a bad alignment). `VerificationResult.FallbackLevel` records the level, and the failures that triggered it are `Error` diagnostics (`verify.V302`, …) followed by a `verify.fallback` warning. The fallback rate is expected to be 0; any non-zero rate is a defect in the stage that produced the failing result. ## 6. JSON `ComparisonJson.Write` / `Read`; schema version 1 in `docs/schema/comparison-v1.json`. The output is compact UTF-8, keys in a fixed order, empty optional members omitted, non-ASCII escaped, so equal comparisons give equal bytes on every platform; a change's `type`, `oldValue`, `newValue` and `substantive` members are optional additions to version 1 (docs/semantics.md) and a reader that does not know them can ignore them; the reader is bounded (`ComparisonJsonLimits`: input bytes, array length, depth) and throws only `ComparisonFormatException`. Reading validates structure and, by default (`verify: true`), re-runs the completeness checks on what it read, so `Comparison.Verification.Passed` is true only when this process proved it; a stored claim is never copied. With `verify: false` the result says `Verification.VerifiedHere == false` and `Passed == false`. A `Comparison` is immutable, so `Check()` only returns the failures (an empty array when everything holds) and never changes `Verification`; to get `Passed == true`, read again with `verify: true`. The exception carries the position of a syntax error (`LineNumber`, `BytePositionInLine`, `Path`). `ComparisonJson.ReadAsync` / `WriteAsync` take streams that refuse synchronous I/O. A source warning (a damaged input) becomes a diagnostic with the code `source.old.` or `source.new.`; the JSON schema is unchanged. ## 7. Plain text `PlainTextSource` reads UTF-8/UTF-16 text (by byte order mark, otherwise UTF-8) into one `Section` holding a `Paragraph` per run of non-blank lines; the lines are kept, joined by LF. It normalizes with the typographic normalizer and gives each paragraph a `TextSpan` anchor into the decoded text and a segment per line. A paragraph that opens with a list marker and goes on (`4.2 Payment`, `(a) the Buyer`; the rule is `ListMarkers.IsMarker`, shared with the PDF and scan sources) is a `ListItem` whose `list.label` is the marker and whose text starts after it, so a renumbering is the same kind of change (a non-substantive attribute change inside a "Renumbering" cascade) whichever kind of file the clauses were read from. Input containing a zero byte (and no UTF-16 mark) is not text. Sources are chosen by content (`DocumentSources.Sniff`), never by file name. ## 8. Extending it - **Alignment:** described in `alignment.md`. A replacement stage returns a normalized `Alignment`; a scorer plugs in through `IPairScorer`. - **In-pair diff and grouping:** produce `LinkDiff`s with replayable `RegionEdit`s; group by emitting `Change`s whose edits are a partition of the link's edits, and `ChangeGroup`s over their ids. Never put data in a group. - **Ledger:** extend `LedgerBuilder` for new categories (page numbers, hyphenation); `Verify` recomputes it from edits. - **Writers:** read `Comparison`; resolve `Anchor`s; use `Change.Old/New` for text and `Groups` for display. - **Sources:** build with `CompareDocumentBuilder`, normalize with `TextNormalization`, attach anchors and segments, set `SourceHash`; implement `IDocumentSource.CanRead` by content. # Alignment What `Stages.Align` does, in the order it does it, so that a different implementation (or a person correcting one) can follow the same rules. The result is an `Alignment` of `Match`, `Move`, `Split` and `Merge` links (see `compare-model.md`); verification checks it afterwards, so these rules decide how a change *reads*, never whether the comparison is complete. Everything is deterministic: the same inputs and settings give the same links. Ties are broken by node number, never by hash order or time. ## 0. Preparation For each node: its compared text (normalization folds whose category is not hidden are undone), its words (letters and digits runs of the word tokenizer, compared exactly), the *shingles* of its words (hashes of neighbouring word pairs; the single words too when the text has three words or fewer), and a *subtree id*. Two nodes have the same subtree id exactly when their kind, compared text, content hash (or property name) and child subtree ids are equal. Block attributes and formatting are not part of the id, so a formatting-only change is still an exact match and shows as a format or attribute change later. Pinned links from the seed alignment are kept first and their nodes are left out of everything below. ## 1. Exact sequence The text core (the settings' algorithm; histogram when the product of the two lengths is over the cost cap) runs over the subtree id plus depth of every unpinned node in document order. Equal runs become `Exact` matches. What lies between two matches is a *stretch*: the old and new nodes in the same stretch are each other's natural partners later. Because the sequence is diffed, a block repeated several times lines up with its own repetitions in order and never crosses. A sequence can have several equally long readings when blocks repeat, and the text core takes one of them without looking at where the blocks sit. Three rules, applied in this order each time the sequence is run, choose among them by structure: 1. **Windows without a stable run.** Between two runs of at least eight matches in step (which nothing could change), the stretch is searched again (a table of at most 1,500 × 1,500 nodes and 2^20 cells) for the longest match, and among the longest the one with the least *drift*: the sum over matches of how far the position of the pair, counted from the window's start, differs between the sides. Of clauses that differ only in a figure, the one in the same place is matched, not the one that would have to jump over its neighbour. 2. **Copies.** When a run of unmatched nodes (or its last or first whole sibling subtrees) is an exact copy of the run next to it, the copies form a chain and the one left unmatched is the one that keeps all the others nearest to their place on the other side (the sum of index differences; ties go to the later copy, because a copy is made after its original). Copies move as whole sibling subtrees, never cut. 3. **Container context.** A *repeated* pair is a match of a block whose subtree id occurs more than once on a side, or whose subtree holds at most five words (a table value, a short label). It contradicts its context when, walking up from both blocks through containers of equal kind, a container is already matched to another node, or is the *provisional partner* of another. Provisional partners are containers (not matched exactly) where the blocks that are not repeated vote: each exactly matched, non-repeated text block votes for every pair of its unmatched ancestors, and two containers are partners when each is the other's single best vote. A pair that contradicts its context is banned: the subtrees on both sides get ids nothing else has, and the sequence is run again (at most four times). So a value in one row is not paired with the equal value of a row it does not belong to, and a deleted row does not lend its equal cell to the row above. Stretches are numbered by the matches before them, from the final sequence. ## 2. Exact relocation (when `DetectMoves`) Subtrees that are still wholly unmatched and equal on both sides are paired wherever they sit, largest subtree first, and among equal subtrees in document order. A moved clause is one link; a moved table is one link for the table plus matches for what is inside it. Subtrees without text or images are never relocated. Two refinements keep equal text from being taken for a move by chance: - Among equal candidates, one that contradicts its container context (step 1, rule 3) is passed over, and one whose parent is the partner of this one's parent is preferred. The first 64 free candidates are looked at. - A single text block is not relocated when both it and its partner each have a close edited partner (similarity at least `MatchThreshold`) among the free blocks of the stretch they sit in: clauses that differ only in a figure, each figure shifted by one place, are edits. The pair is left to step 3, where it competes with those edits. ## 3. Text blocks Eligible blocks are unmatched nodes with at least one word. 1. **Candidates.** An inverted index of shingles (a shingle in more than 24 blocks is ignored) is built for each side. Each block votes for the blocks of the same kind it shares shingles with; the six best (most shared shingles, then nearest position, then lowest number) become candidates, from both directions. Short stretches (at most 100 old-by-new pairs) are compared in full. The work is bounded by the posting cap, so it is near-linear. 2. **Similarity.** The Dice coefficient of the two word multisets; for texts of at most three words also the character-pair Dice, whichever is larger. Pairs under 0.30 are dropped. 3. **Splits and merges** (when `DetectSplitsAndMerges`). A block is a *part* of a longer block (at least four words) when it has at least three words, at least two shingles, and at least 60% of its shingles are in the longer one. Parts of one block that are neighbouring siblings form runs; every window of two to eight parts is scored by the Dice coefficient of the block against the union of the parts. The best window is proposed when it reaches `SplitMergeThreshold` and beats the best single candidate of the block by more than 0.05. A block is not counted as a part of a whole when the whole's previous or next sibling is a better home for it: the sibling is the block's best other candidate, at least `MatchThreshold` similar, and holds a larger share of the block's words than the whole does (or an equal share and a higher similarity). Two neighbouring paragraphs that were both edited, and share template sentences, are then not read as a split. Proposals are accepted best first, skipping any whose nodes are taken. 4. **Judging.** A candidate under 0.95 is ambiguous and goes to the scorer, in batches of 1,024 pairs (`IPairScorer.ScoreBatch`); the rest keep the similarity. The default scorer, `RulesPairScorer`, is the similarity adjusted by structure (+0.02 for each of paragraph style and list level that is equal on both sides, −0.05 for each that differs). A custom scorer replaces that judgement and is never asked about pairs the exact stages or the 0.95 rule settled. 5. **Admission.** A pair in the same stretch needs at least `MatchThreshold`; any other pair is a move candidate and needs `DetectMoves` and either at least `MoveThreshold`, or at least `MatchThreshold` with a *unique partner*: the pair is the best candidate of both blocks and beats every other candidate of either block (local ones included) by at least 0.15. An edited clause that moved has one such partner; two near-identical clauses have two, so neither is guessed. 6. **Choice.** Each admitted pair is weighed: judged similarity, +0.10 if both blocks are in the same stretch, +0.05 if the previous or next sibling of one is matched to the previous or next sibling of the other, +0.02 × (1 − distance between their relative positions in the documents). The global matcher then picks the set of pairs in which no block repeats and the total weight is largest: connected groups of candidates are solved exactly with the Hungarian method up to a size bound, and greedily (heaviest first) beyond it. Chosen pairs in the same stretch have origin `Similar`, the others `Relocated`. A pair that contradicts its container context (step 1, rule 3) and is a repeated pair is left out. ## 4. Containers A container is an unmatched node with no text of its own that groups others (table, row, cell, section, note, header, footer, comment, text box, content control). 1. **By content.** For every matched pair of text blocks or images, each pair of their ancestors of equal kind gets a vote. A pair of containers scores twice its votes over the sum of their content counts (capped at 1); pairs under 0.25 are dropped; the rest are weighed (+0.05 when their parents are paired, +0.02 × nearness) and chosen by the same global matcher. So a table whose rows were reordered and edited is still the same table, and a row with a new column cell is still the same row. 2. **By position.** Walking the paired containers from the roots down: among the children of a pair, the matched children that keep their order are anchors; between two anchors, if the old side and the new side hold the same number (at least one) of unmatched containers of a kind, they are paired in order (score 0.5, origin `Similar`). This pairs empty cells, rows whose text was all rewritten, and the only section or table of a rewritten document. 3. **Text in a paired cell.** In a pair of table cells, between two matched siblings, a single unmatched text block of a kind on each side is paired (score 0.5, origin `Similar`): a cell whose text was rewritten past recognition is one replace, not a delete beside an insert. ## 5. The cost cap `CompareSettings.CostCap` bounds the work of the search for edited blocks: comparisons of two blocks, windows of splits and merges, and the cubic work of exact assignments, with postings visited counted at one sixteenth. When the bound is passed, step 3 stops before it creates any link, a warning diagnostic `align.cap` is added, and the result keeps the exact matches, relocations and the containers: complete, but edited blocks show as deleted and inserted. The exact sequence (step 1) is not capped; it uses Myers while the product of the lengths is within the cap and histogram beyond it (an `align.large` information diagnostic). ## 6. Normalization The links are then normalized (see `compare-model.md`): which one-to-one links are `Move` is decided by the parent and sibling-order rules, not by the steps above. ## Editing `Pin`, `Force` and `Unpin` act on the result. `Stages.Align(..., seed:)` keeps the pinned links of the seed, does not look at its other links, and aligns everything else afresh. # In-pair diff and grouping How the edits inside a pair of aligned blocks are found, and how they become the changes a reader sees. Everything here is deterministic: the same inputs and settings give the same changes in the same order. ## 1. The in-pair diff (`Stages.DiffPairs`) For every link of the alignment (hidden links are skipped), the **joined compared text** of each side is compared. A split or merge joins its nodes with U+001E; a link of one node on each side uses just the node's compared text. Equal text costs nothing. 1. **Words.** Each text is cut at UAX #29 word boundaries (section 4, no tailoring; OneFont's `TextSegmentation`). Segments include white space and punctuation, so they partition the text and every edit maps to an exact character range. Han ideographs and kana are one segment per character (UAX #29 has no dictionary); Thai, Lao, Khmer and Myanmar runs are one segment per run of letters. 2. **Script.** The configured algorithm (`CompareSettings.Algorithm`, Myers by default) diffs the segment sequences. If the number of segment pairs exceeds `CompareSettings.CostCap`, the link falls back to one replacement between the common start and end and a `diffpairs.cap` warning says so. 3. **Cleanup.** The text core's semantic cleanup runs over the script (`DiffCleanup.Semantic`): an equal stretch no longer than the edits on both sides of it is absorbed, so a stray "the" does not cut one rewritten phrase in two, and a lone insertion or deletion slides to a word edge. Cleanup never changes what the script turns into what; the new text and the old text stay exactly reproducible. 4. **Refinement.** A region that replaced one segment by one segment and holds digits on both sides ("1,250,000" → "1,520,000"), and a region in a script without spaces between words on both sides, is diffed again by grapheme clusters, and the refinement is kept if something is shared and it falls into at most three pieces. Other replaced words stay whole: "Parties" replaced by "Party" is one edit, not "ies" by "y". 5. **Formatting.** Text that is equal on both sides can still differ in formatting. The text is cut wherever either side's formatting changes, and each piece is labelled with exactly what differs (`run.bold` off to on, `run.font` A to B). Neighbouring pieces with the same label are one edit. An edit whose old and new text are equal is a **format-only edit**; its `OldRuns` and `NewRuns` carry the formatting on each side. Formatting categories the settings hide never reach this stage; they are in the ledger. The result is a `LinkDiff` per link: ordered, non-overlapping `RegionEdit`s in the coordinates of the joined text, which replay old → new and new → old over text and formatting, plus the attribute edits of the nodes. Verification replays them. ## 2. Change units (`Stages.Group`) The edits are **atomic data** and stay atomic. Grouping chooses **change units**: each unit is one `Change` holding one or more of the link's edits. The units of a link partition its edits, and a unit's sides are ranges of the joined text that **contain** all its edits (a side may show more than its edits, never less). Whole-node insertions and deletions, moves, splits, merges and attribute changes are made as before. `ChangeGroup`s are display regions over change ids and hold no data: one per change, except that a moved node and everything inside it (its attribute changes and text edits) form one region, and so do the cells of one table row that were inserted or deleted whole. Rules, in order, per ordinary link: 1. **Values.** Each text edit is widened to cover every value it touches on each side: a date (`March 3, 2025`, `2026-01-31`, `3/4/2025`, `2025年3月4日`), an amount (`$1,250,000`, `USD 5,000`, `5,000 euros`, with million/billion), a percentage (`12.5%`, `12 percent`), a duration (`30 days`, `10 business days`, `三十日`, `30日`), or a plain number. So "30 days" becoming "60 days" is one change reading `30 days → 60 days`, while its edit is the digit. An edit that only drops or adds characters of a value ("15%" to "5%", "2 500 EUR" to "12 500 EUR") is empty on one side; that side takes in the value it touches when it is the other side's value with exactly the edit's characters left out, so the change reads `15% → 5%` and not "15% deleted". The patterns are simple and deterministic; they are not recognizers (what a value means is decided later). 2. **Overlap.** Units that meet or overlap are joined. A formatting edit that overlaps a unit becomes part of that unit; this can make a unit reach its neighbour, so joining and taking in repeat until nothing changes. 3. **Breaks.** Units never join across a **sentence end** (UAX #29 section 5, on either side's text) or a **clause break**: `;` or `:` followed by white space, or a line break. A unit may still contain an edit that itself crosses a boundary. 4. **Density, per clause** (the run of units between two breaks). With changed words counted on both sides (a word is two half-words; each ideograph, kana or Thai character is one): - at least two units, and at least **a quarter** of the clause's words changed: all the clause's edits become one unit widened to the whole clause on both sides; - at least three units, and at least **12%** changed: likewise; - otherwise units join only when the unchanged text between them is **two words or less**. 5. **Formatting units.** Format-only edits that no text unit took in become format changes (`ChangeKind.Format`): one per edit, and neighbours with the **same label** join across white space alone, if no text unit lies between. A unit's kind is `Format` when all its edits are format-only, otherwise `Insert` when its old side is empty, `Delete` when its new side is empty, else `Replace`. Units are ordered by where they start in the new text. The thresholds (join distance, the two density shares and their minimum unit counts) were chosen on the synthetic dev and train splits and never on the eval split. On dev, boundary F1 moves by at most 0.001 over join distances 0 to 6, whole-clause shares 0.15 to 0.4, scattered shares 0.09 to 0.2 and minimum counts 2 to 4, so the values above are kept; the density rules together are worth about 0.015 boundary F1 (on dev, switching them off scored 0.887 against 0.902, measured before the value rule above and the no-difference rule in docs/measurement.md). Cascades, defined-term renames, typed values and the substantive filter are in docs/semantics.md. They arrange or label the units made here and never change an edit; one thing reaches back into this stage: an edit that replaces a renamed defined term by its new name stays a unit of its own (the *pinned* edits, section 1 of docs/semantics.md), unless the units around it reach it. ## 3. Why the units are not merged edits Merging scattered edits into one big replace edit would be simpler, but it would turn unchanged words into deleted and inserted ones, and a writer that wants the finer view (words struck inside a rewritten sentence) could no longer have it. Keeping the atomic edits in the unit gives both: a writer that shows units reads `Change.Old` / `Change.New`; a writer that shows detail reads `Change.Edits`. ## 4. Verification `Verify` checks every claim here without trusting this code: each edit replays, each change names its link's nodes, its sides' texts and anchors are the source's, its edits lie inside its sides, and every change is in exactly one group. If a check fails, the engine falls back to the reference stages on the same alignment (one change per edit), then to whole-block replacement. # Change semantics What a change is *about*, and which changes belong together. Everything here is deterministic and works without a model. Nothing here alters an edit: cascades and renames are display groups over changes that stay in the list, types and values are labels on changes, and `Verify` checks the same completeness it always did. Where it runs: | Stage | Setting | Does | |---|---|---| | `Stages.Group` | `CollapseCascades` | pins rename edits (section 1), then lifts the changes of a renumbering and of a rename into `GroupKind.Cascade` and `GroupKind.Rename` groups | | `Stages.Classify` | `TypedChanges` for types and values; always for `IsSubstantive` | sets `Change.Type`, `OldValue`, `NewValue`, `IsSubstantive` and the decision | `CompareSettings.Legal` turns both on; `Strict` and `TextOnly` leave them off (Strict collapses and types nothing, so every edit reads as it is). ## 1. Cascades and renames **Cascade.** A *renumbering* is a change whose attribute edits are all a list label going from one value to another (`list.label` `3.2` to `3.3`; surrounding space, brackets and a closing full stop do not count: `3.` and `3`). A *reference change* is a replacement whose old side and new side are each a reference to a clause: a keyword and a number (`Section 3.2`, `§ 3.2`, `Abschnitt 3.2`, `l'article 3.2`, `cláusula 3.2`, `البند 3.2`, `第3.2条`), or just the number when the keyword stands right before it. A reference change is *caused by the renumbering* when its pair of numbers is one of the renumbered labels (`3.2` to `3.3` here and in a label). The cascade group holds every renumbering and every caused reference change of the comparison, one group per comparison, titled "Renumbering". A reference edited by hand to a number no label moved to is not in it; it is a substantive `Reference` change. A reference that the density rules of `docs/grouping.md` merged into a larger unit with other edits is not a reference change and is not in the group either; the other edits are then not hidden by it. **Rename.** The terms a document defines are the quoted phrases that the text goes on to define: followed by `means`, `shall mean`, `has the meaning`, `bezeichnet`, `bedeutet`, `désigne`, `signifie`, `significa`, `是指`, `指`, `とは`, `をいう`, `تعني`, `يعني` and the like, or closed by a bracket as in `(the "Supplier")` and `(以下「X」という)`. Quotes may be `" “ „ « ‹ 「『`. A *renamed term* is a replacement of a term only the old document defines by a term only the new document defines, in at least two places (the definition and a use, or more). A prefix or suffix both texts share (an attached preposition such as Arabic `لـ`) is set aside when that is what it takes to find the two terms. The rename group holds those changes, one group per pair of terms, titled `"old" → "new"`. A change takes part only if its sides are exactly the two terms. **Pinned edits.** To keep a rename from swallowing the other edits of its sentence, an edit that replaces a renamed term by its new name is not joined with the edits around it by the density rules of `docs/grouping.md`; it is its own unit unless the units built without it already reach it (then it goes back among them, as before). The pairs are found from the edits alone, before units are built. A change that sits inside a larger region (the paragraph of a move) is lifted out of that region into its group; the other changes of the region stay together. Every change is in exactly one group after this, as `Verify` requires. ## 2. Types and resolved values `Change.Type` is `None` for an ordinary text change. With `TypedChanges`, in this order: 1. `Formatting` for a format change; `Numbering` for a list-label attribute change (values: the label without closing punctuation). 2. `Typography` for a text change whose sides are equal once white space is dropped and quotation marks and dashes are made plain. 3. In a cascade group, `Reference`; in a rename group, `DefinedTerm` (values: the two terms). 4. A reference outside a cascade is `Reference` with the clause numbers as values. 5. A value change: each side that is present is, as a whole, exactly one value found by the recognizer, and when both are present they are of one kind: `Amount`, `Date`, `Duration`, `Percentage`, `Number`, `Party`. A kind with no resolved value on a side (an ambiguous date) still types the change, with a decision score of 0.5 instead of 1 and no value on that side. 6. A replacement, insertion or deletion of a term the document defines (one side may be empty): `DefinedTerm`. Everything is a rule: `Decision.By` is `Rule`, the source names the rule (`semantics.value`, `semantics.reference`, `semantics.numbering`, `semantics.term`, `semantics.party`, `semantics.format`, `semantics.typography`, or `semantics` for the rest), the score is 1 except as stated in 5. A model that decides later does so with `DecidedBy.Model` and its own score. **Canonical values** are the same in every language: | Type | Canonical | Examples | |---|---|---| | Amount | ISO 4217 code, space, number | `USD 25000` (from `USD 25,000`, `25,000 دولار`), `EUR 18000` (`18.000 EUR`, `2 500 EUR` is `EUR 2500`), `CNY 4800` (`人民币4,800元`), `USD 2000000` (`$2 million`) | | Date | ISO 8601 | `2028-05-16`, `2029-06-01` (`1. Juni 2029`), `2025-01-03` (`3 de enero de 2025`), `2029-08-06` (`6 أغسطس 2029`), `2027-12-27` (`2027年12月27日`); `2025-03` and `--03-16` when the day or the year is missing | | Duration | number, space, unit (`s min h d bd w mo y`) | `30 d`, `10 bd` (business or working days), `18 mo` (`18 Monaten`), `9 mo` (`9个月`), `30 d` (`三十日`), `1 y` (`1年`) | | Percentage | number and `%` | `12.5%`, `6%` (`6 %`) | | Number | invariant decimal | `42` | | Reference | the clause number | `8.4` (from `Section 8.4`, `البند 8.4`, `第8.4条`) | | Numbering | the label without closing punctuation | `2.2` | | DefinedTerm, Party | the text | `Software` | Numbers: a number with one separator and exactly three digits after it (and one to three before, not starting with 0) reads as a thousands group (`1.200` is 1200, `1,200` is 1200, `0.500` is 0.5); in every other case a single separator is the decimal mark; with both, the last is the decimal mark. A date whose day and month could be either (`3/4/2025`; with dots `3.4.2025` is day first) is recognized and not resolved. Month, unit and currency names are written for English, German, French, Spanish, Arabic, Chinese and Japanese; other languages are recognized where they use digits and symbols. The `Party` recognizer finds a name that ends in a legal form (`Inc`, `Ltd`, `LLC`, `plc`, `GmbH`, `AG`, `S.A.`, `SARL`, `B.V.`, `株式会社`, `有限公司` …). ## 3. The recognizer seam ```csharp public interface IValueRecognizer { ImmutableArray Recognize(string text, string? language); } public readonly record struct RecognizedValue(int Start, int Length, ChangeType Type, ResolvedValue? Value); ChangeSet typed = Stages.Classify(changes, myRecognizer); ``` `Recognize` returns the values of a short text (a change side), ordered by position and not overlapping; `Type` is one of the value types (`Amount`, `Date`, `Duration`, `Percentage`, `Number`, `Party`); `Value` is null when the text reads as such a value but cannot be resolved. An implementation is **pure and thread-safe**: no clock, no current culture, no network, no state shared between calls. `language` is a BCP 47 tag when the caller knows it, otherwise null. A recognizer from another package (multilingual, model-backed) replaces `BuiltInValueRecognizer`; only values come from it. References, list labels, defined terms and the substantive flag are decided by the library's own rules, so a recognizer cannot hide or reclassify a change. The recognizer a comparison uses is `CompareSettings.ValueRecognizer` when set, else the one registered process-wide with `ValueRecognizers.Register`, else `BuiltInValueRecognizer`; the reference stages that run when verification falls back read it from the settings too. `Comparison.Of` and the other one-liners therefore use the built-in recognizer unless one of those names another. The `RecursiveAI.OneDiff.Nlp` package supplies one built on OneNLP (`docs/nlp.md`). ## 4. Substantive `Change.IsSubstantive` says whether the change alters what the document says. It is true for a change nobody classified. Not substantive: `Format` changes; `Attribute` changes (style, list label, level); `Typography`; the reference changes of a cascade; whole-paragraph insertions and deletions of empty text; splits and merges whose edits only add or remove the paragraph break and white space. Substantive: everything else, including moves, defined-term changes and renames (a drafting choice a reader still wants to see), insertions and deletions of text, and an image or table inserted or deleted. `Comparison.Changes.Where(c => c.IsSubstantive)` is the filter; the substantive view of the HTML pages and the change report build on it. ## 5. JSON and HTML Schema version 1 gains four optional members on a change (`type`, `oldValue`, `newValue`, `substantive`); see `docs/schema/comparison-v1.json`. They are written only when set (`type` unless `none`, values when present, `substantive` only as `false`), so comparisons made without semantics are byte-identical to before. A reader that does not know them ignores them. Groups of kind `cascade` and `rename` already existed in the schema; a group's `changes` are its occurrences and `title` is `Renumbering` or the term pair. The HTML pages list each cascade and rename group once under "Grouped changes", with its occurrences (`data-group`, `data-occurrence` attributes); the marks in the text are the changes themselves and are untouched. ## 6. Measured Rows and definitions in `docs/measurement.md` (Semantic rows). # Combine, version chains and the change report Normative description of `Combination`, `ComparisonChain` and `ChangeReport` in `RecursiveAI.OneDiff.Documents` (namespace `OneDiff.Documents.Combine`). The compare model, anchors, alignment and grouping they build on are in `compare-model.md`, `alignment.md` and `grouping.md`. ## 1. Combine **Input.** One original document and N ≥ 1 reviewers' copies of it. Reviewer *k* is the *k*-th copy; its index is its position in the list. Every copy is compared with the original by the ordinary stages (`Comparison.Of(original, copy_k, settings)`), so each reviewer has a verified, invertible `Comparison` over the **same** `CompareDocument` object. `Combination.From` accepts ready comparisons and requires that they share that object (node ids are then the same everywhere). **Result.** `Combination` holds the original, the reviewers, their comparisons, the combined changes, the conflicts, diagnostics and a verification record. The combined result never contains a decision between reviewers: nothing is dropped, preferred or merged away unless two reviewers made *the same* change. ### 1.1 Combined changes A *combined change* stands for one change of one reviewer's comparison or, when several reviewers made the same change, for all of those. It carries the first holder's `Change` (both sides, both anchors, edits), the reviewers, the holders (`reviewer`, `change id` in that reviewer's comparison), the reviewers of each edit, and the conflict it is in (or -1). Combined changes are numbered 0..n-1 in the order of the original: by the first old node, then an insertion after the node it follows, then the position inside the node, then reviewer, then the reviewer's own change number. The order is total, so the output is deterministic. ### 1.2 What a change claims Every change makes *claims* about the original, in the original's coordinates. A claim is a kind, the old nodes it concerns, a character range (for text), a key and an effect: | Claim | From | Range / key | Effect | |---|---|---|---| | Text | each edit of a change | old nodes of the link, `[OldStart, OldEnd)` in their joined text (a point when the range is empty) | the new text and its formatting runs (`txt`), or the new formatting only (`fmt`) | | Whole | a node deleted whole | the node | `del` | | Structure | a split or merge | the old nodes | kind, number of new nodes, new text | | Attr | each attribute edit | node and attribute key | the new value | | Moved | a move | the old node; key = destination | `mv` | | Insert | a node inserted whole | key = place: the old node of the linked parent, the old node of the nearest linked sibling before it, the number of unlinked siblings between, and the path below the top of an inserted subtree | node kind, text, attributes, runs | A change with no claim by these rules still claims its old nodes (Structure) or, with none, its place (Insert), so no change can pass unnoticed next to another's. ### 1.3 Identical changes are merged Two reviewers' changes are **the same** when they have the same kind, the same claims (kind, nodes, range, key, effect) and the same new text. They become one combined change attributed to both. Numbers that differ between comparisons (change ids, new-side node ids and offsets) play no part. A reviewer never merges with itself. **Edit-level attribution.** A reviewer's change groups edits into display units, and two reviewers may group the same edit differently (one changed a neighbouring word too). Such changes are not merged, but each edit is attributed on its own: `EditReviewers[i]` lists every reviewer who made the *same edit* as edit *i* of the shown change — the same place in the original, the same new text and formatting — whether or not their change is the one shown. So a shared edit is never mistaken for a private one, and a combined change is never hidden by another. ### 1.4 Conflicts Claims of **different reviewers** conflict when they cannot all be applied to the original. Identical claims never conflict. The rules: | Claims | Conflict when | Kind | |---|---|---| | Whole × anything on a shared node | always (a node deleted by one reviewer and edited, moved, restructured or given an attribute by another) | `EditAndDelete` | | Text × Text, same old nodes | the ranges overlap: they share a character; or a point lies strictly inside the other range; or two points coincide | `Edits` | | Text × Text, different but intersecting old nodes | always (a merge or split edit against an edit of one of its nodes; coordinates are not comparable) | `Edits` | | Moved × Moved, same node | different destinations | `Move` | | Structure × Structure, shared node | different effect | `Structure` | | Attr × Attr, same node and key | different values | `Attribute` | | Insert × Insert | the same place, different content | `Insert` | **Derived claims never conflict.** A list label (`para.label`) that changed, a change of type numbering, and a reference rewritten to follow a renumbering (type reference, not substantive) are consequences of other changes: two reviewers who deleted different clauses renumber the same later clause differently, and merging both deletions gives a third number that neither chose. These changes are attributed and merged like any other and never flagged; the changes that cause them are the ones that conflict, when they do. (The reference rule needs `TypedChanges`; list labels do not.) A point at the edge of a range does not conflict with it; two edits on different words of one paragraph do not conflict. Text × Structure, Text × Attr, Text × Moved, and a moved node's inner edits against its move do not conflict: they commute. A *conflict* is a connected component of conflicting combined changes, with at least two changes of at least two reviewers. Its `Kind` is the most serious (lowest in the table above, by enum order) of its pairs; `Reviewers` lists those involved. Conflicts are numbered in the order of their first change. **Every conflicting version is shown; none is preferred.** A change shared by reviewers A and B that overlaps a change of C puts A's, B's and C's versions in one conflict. ### 1.5 Verification `Combination.Check()` returns failures with codes: | Code | Check | |---|---| | C001, C002 | each reviewer's comparison was made from the combination's original and passes `Comparison.Check` | | C101–C104 | combined change ids are 0..n-1; each has holders; every holder exists; no reviewer change is attributed twice | | C105 | every holder's change equals the shown change in kind, old side, edits (place, new text, formatting) and new text | | C106, C107, C108 | the reviewers equal the holders' reviewers; edit attributions are as many as edits, include every holder, and name only reviewers who made that edit | | C109 | every change of every reviewer is in some combined change | | C201–C204 | the stored conflicts equal the conflicts recomputed from the changes (ids, kinds, members, reviewers), each has ≥2 changes of ≥2 reviewers, and each change's conflict id is right | C101–C104 and C109 together say that the changes attributed to reviewer *k* are exactly the changes of *k*'s comparison, each once. Projecting the combined result onto *k* (`ChangesOf(k)`, each by its holder entry for *k*) therefore gives the very change list of *k*'s verified comparison, so old + changes = *k*'s copy and the copy − changes = old, per reviewer, on every run. **Fallback.** If a check fails, the combination is rebuilt without merging (every reviewer's change listed on its own, conflicts recomputed) with a `combine.fallback` warning and `combine.C…` error diagnostics for the failures that were recovered from (`Verification.FallbackLevel` 1). If it still fails, `ComparisonVerificationException`. With `CompareSettings.ThrowOnFailure` (strict) the first failure throws. Without the model, nothing here depends on it. ### 1.6 Entry points `Combination.Of(original, copies, settings, scorer, token)`; `Combination.From(comparisons, token)`; `Redline.Combine(originalPath, copyPaths, outputPath, settings)` (thin: reads the files, `Combination.Of`, `SaveAs`); CLI `onediff combine ... -o `. `Combination.SaveAs` writes the change report (`.html`) or the JSON form (`.json`). Per-reviewer inline redline documents and DOCX with revision authors per reviewer come with the Word writer. ## 2. Version chains `ComparisonChain.Of(versions, settings, scorer, token)` compares version *i* with *i* + 1 for every step and the first version with the last (the *net* comparison). Step *i*'s new document is step *i* + 1's old document: the same object, so the steps compose. Every step and the net comparison are ordinary verified comparisons; `Check()` re-runs them (H001–H004) and confirms the steps meet and the net runs from the first version to the last. **Step attribution.** Each change belongs to the step that made it (a step's changes are its own comparison's). For the net view, `StepsOf(netChangeId)` lists the steps that touched a net change. A step touches it when 1. one of its changes concerns a block the net change's old blocks became (followed forward through the steps' alignments), or that the net change's new blocks came from (followed backward); 2. one of those blocks is deleted or created in that step (a block removed with its parent has no change of its own); 3. for a net move: the step moved anything (moving one block changes the relative order of the others, so the net comparison may show a different block as moved); 4. otherwise (the net comparison paired interchangeable blocks, say empty cells, differently from the steps): the steps that made a change of the same kind, and failing that any step that changed anything or reordered blocks (identical blocks, such as empty cells, can swap places with no change to show). The grain is the block: two edits to one paragraph in different steps are both attributed to both steps in the net view; the step comparisons say exactly which step made which edit. Entry points: `ComparisonChain.Of`, `Redline.Chain(versionPaths, outputPath, settings)`, CLI `onediff chain ... -o `. ## 3. Change report `ChangeReport.From(comparison | combination | chain)` is a plain data object; `ToHtml()` renders it as one self-contained page (inline style, no scripts, no time or environment: the same report gives the same bytes). - **Counts** by change kind and by change type (only those that occur), the number of substantive changes, and the ledger (hidden differences by category). - **The substantive list** (`IsSubstantive`): location, kind, the old and new text (shortened to 200 characters), and the resolved values (`Duration 30 d → 60 d`). *Location* is `§` and the list label of the block or its nearest labelled ancestor (`§4.2`), otherwise `¶` and the block's number among paragraphs and list items. - **Combination:** changes attributed per reviewer, how many were made by one, two, … reviewers, and a section per conflict listing every reviewer's version side by side with the reviewers named. - **Chain:** a section per step with its counts and substantive list, then the net section, each net change with the steps that touched it. ## 4. JSON forms Both embed complete `onediff.comparison` documents (see `schema/comparison-v1.json`, unchanged) and add attribution. Schemas: `schema/combination-v1.json`, `schema/chain-v1.json`. Written with `Utf8JsonWriter`; byte-identical for equal inputs. `onediff.combination` v1: `original` {name, hash}; `reviewers` [{index, name, hash}]; `verification` {passed, fallbackLevel, failureCount}; `changes` [{id, kind, reviewers [], holders [{reviewer, change}], editReviewers [[…]] (omitted without edits), conflict (omitted when none)}]; `conflicts` [{id, kind (`editAndDelete`, `edits`, `move`, `structure`, `attribute`, `insert`), changes [], reviewers []}]; `diagnostics`; `comparisons` [full comparison per reviewer, by index]. A combined change's content is the holder's change in `comparisons[reviewer].changes[change]`. `onediff.chain` v1: `versions` [{name, hash}]; `steps` [{index, from, to, comparison}]; `net` {comparison, changeSteps [{change, steps []}]}; `diagnostics`. The existing `onediff.comparison` document is unchanged. There is no per-change step or reviewer member in it, because a comparison has one old and one new side: attribution belongs to the document that holds several of them. ## 5. Limits and cost Claims are grouped by old node and by insertion place; pairs are compared only inside a group, so cost is quadratic only in the edits to one node. The combination's own work is a small fraction of the N comparisons it is built on (see `measurement.md`). Memory is linear in the changes. Cancellation is observed between comparisons. ## 6. Samples `docs/samples/inputs/` holds the plain-text contract used below: an original, three reviewers' copies (`alice`, `bob`, `carol`) and four versions (`v1` to `v4`). `docs/samples/combine-report.html` is `onediff combine original.txt alice.txt bob.txt carol.txt -o combine-report.html` (two shared changes, one conflict over a payment period that two reviewers set to 60 and one to 30 days), and `docs/samples/chain-report.html` is `onediff chain v1.txt v2.txt v3.txt v4.txt -o chain-report.html`. # HTML output and the redline entry points This note describes what `RecursiveAI.OneDiff.Documents` writes as HTML, how a page maps back to the comparison it came from, and how adapter packages plug their inputs and output forms into `Redline.Create` and `Comparison.SaveAs`. ## 1. The page Both layouts are one self-contained file: UTF-8 without a byte order mark, LF line ends, inline CSS, no scripts, no external fetches (no `src`, no `url()`, no `@import`, no link other than in-page `#` anchors), system font stacks. The same comparison and style give the same bytes on every platform; the clock is never read. A timestamp appears only when `RedlineStyle.Timestamp` is set. ``` RedlineStyle.Language ("en"); the page furniture is English charset, viewport, generator, , <style> <body class="inline|sbs"> <header class="cover|top"> <h1>; with CoverPage: file names, settings, hidden categories, completeness line, counts by change type; then the key <main aria-label="Redline"> the layout below <section class="ledger"> "Differences not shown": counts per category, first 50 locations each <section class="notes"> warnings and errors from the stages (only when there are any) <footer> "Compared by OneDiff x.y.z", input names with SHA-256, timestamp if given ``` **Style as data.** The sheet is generated from the `RedlineStyle`: each mark (insertion, deletion, move from/to, formatting) becomes colour plus a `text-decoration` (`underline`, `underline double`, `line-through`, `line-through double`) or a highlight (the colour mixed 78% with white behind near-black text, and a 2 px bottom rule in the full colour). Glyphs are written into the markup (`<span class="gl" aria-hidden="true">`), so a change never depends on colour alone where the style gives a glyph. Change bars are a left border in the margin (`ChangeBars`, `ChangeBarColor`). `CoverPage`, `Footer`, `TypeTags`, `Title` and `Language` do what their names say. Colours are checked to be `#RRGGBB`; anything else is replaced by black, so a style from an untrusted source cannot reach the sheet. **Presets.** Blackline (the default): blue double-underlined insertions, red struck deletions, green double-struck/double-underlined moves, bars, cover page, footer. SideBySide: colour-blind-safe blue (#0072B2) and orange (#E69F00) highlights with `+`, `−`, `→`, `←` glyphs; no bars, cover page or footer; the side-by-side form picks it when the settings still hold the default blackline. Word: red underlined insertions and struck deletions, green moves, a bar. Developer: green and red backgrounds with `+`/`-`, purple moves. **Contrast of the preset colours on white** (WCAG 2.x ratios; 4.5 is AA for body text, 3 for large text and graphics). Blackline: insertion 8.59, deletion 4.00, moves 5.14, formatting 4.00. Word: insertions/deletions/formatting 6.48, moves 2.87. SideBySide and Developer put near-black text (`#1a1a1a`) on the tint, 12.2–14.6, so the mark colours (SideBySide orange 2.25 and grey 2.85 against white) are only the rule under the highlight and the glyph, never the text. Where text itself takes the mark colour and falls below 4.5 (Blackline deletion and formatting, Word moves), the mark is also a decoration and hidden text names it, so the information is not carried by colour; the values are in the style presets, not the writer. ## 2. Inline layout The new document's tree in order. Each node is a `div.blk.k-<kind>` (a real `table`/`tr`/`td` for tables) with `data-new` (new node id), `data-old` (the linked old node ids, space separated) and `data-link` (the alignment link). Its own text is one `p.t` (`white-space: pre-wrap`), a list label is a `span.lbl` beside it, other kinds get a `div.kl` label (Footnote, Image, …). Children nest inside the block. Text inside `p.t`: | Markup | Meaning | |---|---| | bare text | unchanged | | `<ins data-change=N>` | inserted text of change N | | `<del data-change=N>` | deleted text of change N, in place | | `<span class="fmt" data-change=N title=…>` | text whose formatting changed (the title says how) | | `<span class="mv to|from" data-role data-change id=mvt-N|mvf-N>` | the new / old place of a moved block, with an `a.mvlink` to the other place | | `<span class="sep">¶</span>` inside `del` | a node boundary inside deleted text (a merge) | | `<span class="pbrk">¶</span>` inside `ins`/`del` | a paragraph break that was inserted/removed (a split/merge at a block boundary) | | `<span class="ctl" data-cp=hex>` | a control character or unpaired surrogate HTML cannot carry | | `<span class="sr">`, `<span class="gl">`, `<span class="tag">` | hidden screen-reader label, glyph, type tag: not document text | Deleted blocks and the old place of moved blocks appear as `div.blk.ghost` (with `data-old`) where they used to be: before the next kept block, and ahead of blocks inserted just before it, so a removed block reads before its replacement. Attribute changes (style, list label, hashes) are `div.attrs > span.ae[data-change data-key data-old data-new]` under the block, shown as a label and unquoted values (`List number: 1.5 → 1.2`, `(none)` for a missing side); the machine-readable key and values stay in the attributes. Hidden content (comments, properties under the Legal settings) is not drawn; it is in the ledger section. A change that belongs to a link whose other side the settings hide is a ledger entry, not an element. ## 3. Side-by-side layout `table.sbs`: header cells "Old: name" / "New: name", one row per block in the merged order. A kept block is on both sides; deleted blocks only left, inserted only right, a moved block twice (old place left with `mv from`, new place right with `mv to`). A split or merge is one row with all its blocks. The left cell holds only unchanged and deleted text, the right only unchanged and inserted text; a formatting change shows on both sides. Container blocks with no text and no change get no row. Indentation is the tree depth (`d1`…`d5`). ## 4. How a page reads back The tests parse every page they write and check, per comparison and per preset and layout: 1. the text of each `p.t` with `ins` removed (and `del` kept) equals the old node's compared text; with `del` removed it equals the new node's — for every visible node, including splits and merges (blocks joined by U+001E) and ghosts; 2. every change is exactly the elements its edits call for: whole-node insert/delete = one `ins`/`del` with the change's text; edits = `ins`/`del`/`fmt` elements whose concatenated text equals the edits' new/old/formatted text (side-by-side drops U+001E); a move = one `from` and one `to`; each attribute edit = one `ae` with its key and values; no element refers to an unknown change; 3. hidden nodes are absent; nothing in the page can run or fetch anything. "Compared text" is the text under the settings (`ProjectedGroup`): hidden normalizations (typographic quotes, dashes, whitespace under Legal) stay folded and are counted in the ledger section rather than drawn. HTML validity is checked separately with the W3C Nu validator. ## 5. Escaping All text from documents, names, hashes, ledger details and diagnostics goes through one escaper. Text: `&`, `<`, `>` are entities; characters HTML cannot carry (C0 controls other than tab and LF, DEL and C1 controls, unpaired surrogates, noncharacters) become `span.ctl` with the code point in `data-cp` and a visible control-picture glyph, so reading the page back gives the original characters. Attribute values: `&`, `"`, `<`, `>` are entities and the uncarriable characters become U+FFFD. `Language` must be letters, digits and hyphens or it falls back to `en`. ## 6. Entry points and the registration seam ```csharp Redline.Create("v1.txt", "v2.txt", "redline.html"); // form from the extension Redline.Create("v1.txt", "v2.txt", "side.html", RedlineForm.HtmlSideBySide); var cmp = Comparison.Of("v1.txt", "v2.txt", CompareSettings.Legal); // inputs sniffed by content cmp.SaveAs("redline.html"); // settings' style cmp.SaveAs("redline.json"); cmp.SaveAs("redline.html", RedlineStyle.Developer); byte[] bytes = cmp.ToBytes(RedlineForm.Html); // or cmp.Save(stream, form) ``` `RedlineForm` is a small abstract class (name, extensions, preferred style, `Write`). A form passed explicitly needs no registration. `DocumentSources` and `Redline` hold two process-wide lists, filled by explicit calls and never by discovery: - `DocumentSources.Register(IDocumentSource, priority = 0)` (or `Redline.Register(source, priority)`): sources consulted before the built-in plain-text one, highest priority first and the newest first among equals; same `Name` replaces. A specialist source (the scan source) registers above the source that would also accept its input, so the order of `Register` calls does not matter. - `Redline.Register(RedlineForm)`: forms selected by extension; the latest registration of an extension wins; same `Name` replaces. Both are thread-safe and idempotent. An adapter package exposes one `Register()` that makes those calls. What a user writes: ```csharp PdfRedline.Register(); // once at startup; each adapter package has one Redline.Create("a.pdf", "b.pdf", "out.pdf"); ``` Without that call the failure names the package: `No output form is registered for '.pdf'. Reference RecursiveAI.OneDiff.Pdf and call its Register method once at startup.`, and an unreadable PDF input says it looks like a PDF and names the same package. Code that wants no global state uses `CompareDocument.From(input, source)`, `Comparison.Of(oldDocument, newDocument)` and a form's `Write` directly. A module initializer was rejected: it would run only after the adapter assembly is touched, which is exactly what `Redline.Create("a.pdf", …)` does not do, and library module initializers are discouraged for trimming and AOT. When no style is given: the form's `PreferredStyle` (side-by-side: `SideBySide`) if the settings' style is still the default blackline, otherwise the settings' style. ## 7. Command line `onediff compare <old> <new> <output> [--form html|html-side-by-side|json] [--settings legal|strict|text] [--style blackline|sidebyside|word|developer] [--title T] [--language tag] [--timestamp ISO-8601] [--no-cover] [--no-footer] [--no-bars] [--type-tags]` over `Redline.Create`. ## 8. Samples `docs/samples/<preset>-inline.html` and `<preset>-side-by-side.html` for the four presets, written from a short pair of documents made for the purpose (a replaced number, a deleted clause, an inserted clause, a moved clause that renumbers, a formatting change, a typographic difference in the ledger). A test compares them with what the writers produce now; `ONEDIFF_UPDATE_SAMPLES=1` rewrites them.  # PDF comparison How `RecursiveAI.OneDiff.Pdf` reads a digital PDF into the compare model, and how it marks a comparison on the original pages. Everything stated here is checked by the package's tests. ## 1. Reading a PDF `PdfDocumentSource` recognizes a PDF by its `%PDF-` header within the first 1024 bytes (ISO 32000-2 7.5.2), never by file name. It reads the text with OnePDF's extraction (glyphs, words, lines, blocks in reading order, header and footer roles) and builds the flowing compare model. ### Structure | Source content | Node | |---|---| | Body block | `Paragraph`; a block that starts with a bullet, a number or a letter marker (`•`, `-`, `1.`, `(a)`, `iv)`, or a dotted-decimal clause number such as `1.1` or `10.2.3.`) is a `ListItem` with the marker as the `list.label` attribute and the rest as its text | | A marker that the text analysis made a block of its own, beside a block on the same baseline | the same `ListItem` | | Block recognized as a table | `Table` → `TableRow` (a line) → `TableCell` (the words between two cell starts) → `Paragraph` | | Running headers and footers | a second `Section` after the body: one `Header` and one `Footer` node for a page whose header or footer text differs from the previous page's, holding its paragraphs. Pages that repeat the text add nothing, so a document with more pages does not gain a header per page | Body paragraphs are joined where layout cut them: - **Across pages:** the last paragraph of a page and the first of the next are one paragraph when the first does not end a sentence (`. ! ? : ;` and their CJK, Arabic and Devanagari forms, looking through closing quotes and brackets) and the second starts with a letter that is not upper case. - **Across columns:** within a page, a block that starts higher up the page than the previous one ended continues it under the same rule. - **Split-off lines:** a block that starts closer under the previous line than any paragraph gap (less than 0.4 line heights) continues it under the same rule. The text analysis splits a justified first line off this way. - A list item is continued the same way; a table never is. - **Grids the text analysis did not flag as tables:** when it returns each cell of a grid as a block of its own, consecutive short blocks (at most three lines and eight words each, not list markers) that fall into rows of the same number of cells (two or more), each cell overlapping the one above it, for at least two rows, become a `Table` of `TableRow` and `TableCell` nodes. A block right under a cell of the row (closer than half a line height) and overlapping it is that cell's next line, because a narrow cell wraps ("1 200" and "EUR"). Any block or row that breaks the pattern ends the grid; a grid with fewer than two complete rows stays paragraphs. The lines of one cell are joined by a space, not a line break (a word broken by a hyphen keeps its break, so the hyphenation fold joins it). A grid that opens a page, under the table that closed the page before, goes on in that table when it has as many columns, and a row it repeats as a heading (the same cell texts as the table's first row) is not a row. A first line of a block that is a running header's text, or a line alone at the top of a page, is left out of the grid reading, because a header right above a repeated heading can be read as part of the heading's first cell. - A clause number that opens a wrapped line inside a block starts an item only after a line that ended a sentence or is three words or fewer, because a reference to a clause ("see clause\n2.2.") can fall there. Reading the clause number as the label gives a Word file's `ListItem` and a PDF's the same shape, which is what lets a Word/PDF pair align. A decimal that starts a line (`3.5 million`) is read as a label too; that is the cost of the rule. Not in the model: images, vector graphics, annotations, form fields and document properties. ### Formatting Each word carries the formatting of its first glyph: `run.bold` (font name contains bold, black, heavy, semibold or demi), `run.italic` (italic or oblique), `run.font` (the font name up to the first `-` or `,`), `run.size` (points, rounded to half a point) and `run.color` (`#RRGGBB`, left out for black). A run goes on over the space to a following word with the same formatting. ### Ledger folds | Fold | Category | Where | |---|---|---| | line break inside a paragraph → one space (nothing between two characters of a script written without spaces: CJK, fullwidth forms, Thai) | `LineBreaks` | body and furniture | | hyphen (`-`, U+2010, soft hyphen) at the end of a line, preceded by a letter, followed by a lower-case letter on the next line → nothing | `Hyphenation` | body and furniture | | number equal to the page's number, the page's label or the page count, in a header or footer → `#` | `PageNumbers` | furniture | | quotes, dashes, runs of spaces | as for every source | body and furniture | Each fold keeps the original characters, so settings that do not hide a category show its differences as ordinary changes. ### Anchors A node's anchor has one quadrilateral per line (for a list item, the line includes the marker). For any other block it is the hull of the block's own words, one box per run of neighbouring words: a run ends where the line changes, the reading direction turns, or a gap wider than two line heights opens, because whatever lies in such a gap (another table cell or column) is not the block's. Its text is covered by segments, one per word, one per punctuation mark and one per space between two words on a line (cut as the text core's word tokenizer cuts; a space at a line or page break, or across a gap wider than two line heights, has none), each with one quadrilateral: **a change found by a word diff therefore begins and ends on segment boundaries, and its anchor covers exactly the changed text**. Quadrilaterals use the coordinates OnePDF reports: points, **upright page space** (origin at the lower left of the page as displayed, y up, the page's `/Rotate` and crop box applied; ISO 32000-2 7.7.3.3), a zero-based page index, and the corners in the order **lower left, lower right, upper right, upper left** of the text. A token's quadrilateral comes from its glyphs when the glyphs spell the word (left-to-right, horizontal text), otherwise from sharing the word's box evenly along the writing direction. In right-to-left lines the glyph order is not stated precisely enough to rely on, so the order whose glyph texts spell the word is found by trying the plausible ones. **Overlapping glyphs.** Glyph boxes of different words, or of other characters of the same word, can overlap on the page (a line of underscores running under the next word, a line of stars with text printed over it). A token's box is cut back, midway between its own outermost glyph centre and the foreign glyph's centre, until no glyph that is not the token's own has its centre inside it; a glyph from the line above or below is cut off vertically. Where a foreign glyph lies between two of the token's own glyphs, no box can leave it out, and a highlight takes it with them: the page genuinely has two texts in one place. ### Errors and limits `Read` throws `InvalidDataException` for an input it cannot read; `PdfSourceErrors.FailureOf` returns why: `Malformed`, `Encrypted` (no password, or a wrong one; `PdfSourceOptions.Password` supplies it) or `TooLarge` (`MaxPages` default 10,000; `MaxCharacters` default 64 million; OnePDF's own limits). A truncated or damaged file that OnePDF can repair is read as repaired. Cancellation throws `OperationCanceledException`. ## 2. Original-page markup `PdfRedline` (a sealed class and a `RedlineForm`; preset `PdfRedline.OnPages`) writes real markup annotations (ISO 32000-2 12.5.6.10 to 12.5.6.14) onto the pages of the new PDF with `MarkNewDocument`, and optionally onto the old one with `MarkOldDocument`. The PDF's own bytes must be the input the comparison read (their SHA-256 is checked against the document's source hash). Page content is never touched. | Change | On the new document | On the old document | |---|---|---| | Insert | Highlight, insertion colour | – | | Replace, Split, Merge | Highlight, insertion colour; pop-up holds the old text | StrikeOut, deletion colour; pop-up holds the new text | | Delete | Caret where the text was, deletion colour; pop-up holds the old text | StrikeOut | | Move | Highlight, move-to colour; pop-up holds the old text | Highlight, move-from colour | | Format | Underline, formatting colour; the first line names the old and new formatting | Underline | | Attribute (list label) | Squiggly, formatting colour; the first line names the attribute and both values | Squiggly | - **Colours and glyphs** come from `PdfRedline.Style`, default `RedlineStyle.SideBySide` (blue insertions, orange deletions, `+` and `−`). - **Where a caret goes.** For a deletion inside a paragraph, at the end of the text before it (or the start of the paragraph's text when the deletion is first). For a deleted paragraph, after the last text before it that has a place in the new document, or at the start of the document's text when there is none. - **Contents.** The first line names the change and starts with the style's glyph (`+ Inserted`, `+ Replaced`, `− Deleted`, `← Moved`, …); the text after the first line feed is the other side's text exactly (`PdfRedline.SplitContents`). `/T` is `Author` (default `OneDiff`). - **Names.** `/NM` is `onediff-c{change id}-p{page}`, with `.{n}` for further annotations of the same change on the same page (a change of more than 2048 quadrilaterals). A change spanning pages gets one annotation per page. Every annotation maps to one change; every change has annotations or is listed in `PdfMarkupResult.Skipped`. The only skipped changes are those of containers with no text of their own (a section, a table, a row), whose content carries its own changes. - **Quadrilaterals** on one line that follow one another are merged, so a phrase is one band. - **Pop-ups** sit to the right of the marked area when the page has room, else to its left, closed. - **Rotation.** Anchors are in upright space; the writer maps them to default user space through the page's crop box and `/Rotate`. - **Determinism.** The same comparison, PDF and settings give the same bytes on every platform. `/M` is written only when `ModifiedDate` is set; nothing reads the clock. Identifiers come from the change ids and OnePDF's hash-derived `/ID`. - **Saving.** Incremental by default: the output is the original bytes followed by an update, so signatures over the original stay in place where their permissions allow markup (12.8.2.2). A file that was rebuilt on opening cannot take an update and is rewritten whole; `PdfMarkupResult.SavedIncrementally` says which happened. `Incremental = false` always rewrites. - **Encrypted input.** Needs `PdfRedline.Password`; the output keeps the input's encryption. - **In the browser (WebAssembly).** The .NET browser runtime has no AES, so encrypted PDFs cannot be read or marked there; reading reports `Encrypted`. Everything else works in the browser. ### As an output form `PdfRedline.Register()` registers `PdfDocumentSource` and `PdfRedline.Auto` (name `pdf-auto`, extension `.pdf`) with the `Redline` statics, once and idempotently. **`Auto` follows the new document, and only the comparison** [Agent default]: when the new document's `SourceFormat` is `application/pdf` (or a scanned image, `image/png`, `image/jpeg`, `image/tiff`), it writes `PdfRedline.OnPages` (the new document with the marks, with the `SideBySide` look unless the caller or the settings chose one); for every other new document (text, Word) it writes the rendered redline of section 4. So `Redline.Create("a.pdf", "b.pdf", "out.pdf")` marks the new PDF's pages, `Redline.Create("a.txt", "b.txt", "out.pdf")` renders a blackline, and a pair with a PDF as the new document and a text file as the old one is marked on the PDF. Both forms stay selectable by name: `cmp.SaveAs("out.pdf", PdfRedline.Rendered)` renders a PDF pair, and `PdfRedline.OnPages` on a comparison whose new document is not a PDF throws. No registration order, no process state and no earlier call decides which one you get. The marking form needs the new document's bytes. They are kept on the document itself (`CompareDocument.Origin`, set by `PdfDocumentSource` and the scan source), in memory only: not hashed, not compared and never written to JSON. A comparison whose new document has no origin (one read back from JSON, or one built by hand) cannot be marked through `SaveAs`; `Auto` and `OnPages` throw `InvalidOperationException` saying so and naming the two ways out: `PdfRedline.OnPages.MarkNewDocument(comparison, bytes, stream)` with the PDF's bytes, or `PdfRedline.Rendered`. The style passed to `SaveAs` replaces `Style`. **Pages without text, and damaged files.** A PDF whose pages carry no text at all is a scan (or a drawing), and reading it as text would compare an empty document. `PdfDocumentSource` refuses it with an `InvalidDataException` whose `PdfSourceErrors.FailureOf` is `NoTextLayer` and whose message names `RecursiveAI.OneDiff.Ocr` and `OcrRedline.Register()`; the refusal does not depend on the settings. A PDF whose pages have no content stream at all (no text, and no image to read by OCR either) is not a scan but a truncated or damaged file: it is refused as `Malformed` with a message saying so, with or without the OCR package registered. A file from which no page could be read is `Malformed`. A file that opens but from which OnePDF could not recover some content comes with a warning on the document, which the comparison reports as the diagnostic `source.old.pdf.unrecovered` or `source.new.pdf.unrecovered` (and `CompareSettings.Strict` throws on it); repairs alone are an informational `pdf.repaired` diagnostic. ### Whole words on the page A word diff can end inside a word (a character-level refinement of a number or an identifier), and the text core reads some runs as one word (`10.8`, `a_b`). The page marks **whole tokens of the text core's word tokenizer**: a changed range that cuts a token is rounded out to the token. The other side's text in the pop-up is exact. A change that is only white space across a line break has no box of its own and is anchored on its block; the check has no text to compare for it. ## 3. Reading the marks back The tests re-extract every marked PDF with OnePDF and check, for every pair: each annotation has a change; each change has its annotations or a skip reason; the text under the quadrilaterals of the annotations of a change equals the change's new text (spaces, line-end hyphens and typographic variants aside, and with folded page numbers read as the digits the page shows); each pop-up's text after the first line equals the change's old text. The text under a change that cuts a token is compared with the change's text rounded out to the token, taken from the text as the page shows it. Where glyph boxes of different words overlap on the page and a highlight cannot avoid taking the other word (see Overlapping glyphs), the text under it may also be given with the overlapping words left out; the check accepts that, and only that, as equal. The gate set is run by `PdfGateRunTests` (environment `ONEDIFF_PDF_GATE_DIR`, `_FROM`, `_COUNT`, `_JOBS`, `_OUT`) over every pair, with the Legal and the Strict settings: verification passed with no fallback and a clean re-check; the marks re-extracted as above on the new document, and the old document's marks likewise; a second run giving byte-identical comparison JSON and marked PDFs. ## 4. Rendered redline `PdfRedline.Rendered` (inline) and `PdfRedline.RenderedSideBySide` are `RenderedPdfRedline` forms: a new PDF laid out from the comparison alone, for any kind of input, plain text included. The package builds the same page as the HTML redline (`docs/output-html.md`: cover page, key, marks, ledger, footer with the input names and hashes) with a style sheet for paper, and renders it with OneRender's `HtmlPdf.ToPdf`. `Render(comparison, stream, style)` is `Write` plus a result with the page count and the renderer's notes. - **Layout.** A4 (`Paper = PdfPaper.Letter` for Letter: `new RenderedPdfRedline { Paper = PdfPaper.Letter }`), margins 18 mm and 16 mm, the running page number in the bottom margin. Side by side is landscape: one grid row per block, old on the left and new on the right, and a row is kept whole on a page when it fits on one (see the limits below). The cover page, change bars and footer follow the style (`CoverPage`, `ChangeBars`, `Footer`); the timestamp only when the style carries one. Each block takes its direction from its first strong character (`unicode-bidi: plaintext`). - **Look.** The comparison settings' style unless the caller passes one; `Blackline` for the inline layout and `SideBySide` for the side-by-side layout when the settings left the style at the default. Marks are drawn with the style's colours, decorations (underline, double underline, strike-through, highlight) and glyphs. A control character the page shows as a control picture takes the colour of the change it is in. - **Static.** The page uses no script, no network and no font outside the ones the renderer carries (Liberation Sans). The text-for-screen-readers labels of the HTML page ("inserted:", "deleted:") are not written, since the colour, decoration and glyph of the style carry the meaning and the key explains them. The page's closed ledger sections are written open. - **Scripts.** The renderer draws a script only when a font that covers it is referenced by the application: `RecursiveAI.OneRender.Fonts.Cjk`, `.Fonts.Noto.Arabic`, `.Fonts.Noto.Devanagari` and the other `.Fonts.Noto.<Script>` packages, `.Fonts.Emoji`. Each package registers itself when the application references it; nothing in `RecursiveAI.OneDiff.Pdf` changes. Without a font the characters are not drawn but stay in the text layer, so extraction and search still find them (the renderer's notes do not say so; see the limits). - **Tagged.** The output is a tagged PDF with the document language (`RedlineStyle.Language`) and title (`RedlineStyle.Title`, or "Comparison of *old* and *new*") set; headings, paragraphs, the counts and ledger tables and the lists carry the renderer's structure types. - **Deterministic.** The same comparison and style give the same bytes: no creation date is written and the clock is never read. - **Limits** (renderer 1.3.3; `OneRenderGapTests` pins what is pinned with a minimal case): - a table row whose cells differ in height can lose a line at a later page break (a 40-line cell beside a 100-line cell lost its 97th line). The side-by-side page therefore keeps each row whole where it fits on a page and does not repeat the column headings on each page; a row taller than a page can still lose a line at its cut; - a page list holds a few million glyphs: a redline of that size (the largest text pair of the test corpus, 29 MB) fails with an `InvalidOperationException` that names the forms without the limit (HTML, JSON, markup on the original pages); - a colour emoji needs `RecursiveAI.OneRender.Fonts.Emoji` referenced, like every other script. ### Fonts for other scripts `RecursiveAI.OneDiff.Pdf` and the `onediff` tool reference no script font package, so only Latin (Liberation Sans, which the renderer carries) is drawn; the characters of other scripts stay in the text layer but are not drawn, and the renderer's notes say so (`RenderedPdfResult.Notes`, one line per missing glyph run). To draw them, reference the font packages the documents need in the application, for example `RecursiveAI.OneRender.Fonts.Cjk`, `RecursiveAI.OneRender.Fonts.Noto.Arabic`, `RecursiveAI.OneRender.Fonts.Noto.Devanagari`; each registers itself, and nothing else changes. The tests reference those three. ### Bidirectional controls The PDF source keeps the extraction as it comes, including U+200C/U+200D and the bidirectional control characters, and does not set `PdfTextOptions.RemoveBidiControls`: the extractor reads right-to-left runs in logical order, and the comparisons made with the controls kept verify and show no changes that are only controls on the 1,110 test pairs. ### Reading the rendered pages back `RenderedGate` (in the tests) re-extracts a rendered PDF with OnePDF and checks it against the comparison. For an inline page it puts every glyph in a class by its fill colour (inserted, deleted, moved from, moved to, formatting, plain; a style whose colours differ is needed, and the gate uses the blackline with its own colour for moves from). Without the classes that are not in the new document it must hold the new text, and without those not in the old document the old text, in order, block by block; each edit's inserted text must be in the inserted class and its deleted text in the deleted class. For a side-by-side page the left half must hold the old blocks and the right half the new blocks. Text is compared with whitespace and joiners left out, as the page shows it (control characters as their control pictures, the folds of categories that are not hidden undone). Lines are read in the order a reader takes them: bands cut at the vertical gaps, columns cut at horizontal gaps that a line from the other side does not cross (the cells of a table row one after another), and a side-by-side page column by column. A document with right-to-left text anywhere is still checked as a bag of characters, because the neutral characters between mixed-direction runs depend on the bidirectional algorithm. What the extraction cannot see: underlines, strike-throughs and highlights are drawn as shapes, so a style whose marks differ only by decoration (the Word preset's insertions and deletions are both red) is checked for text only; the page's colour is the only styling it reads. The text corpus and the PDF pairs are run by `RenderedGateRunTests` (`ONEDIFF_RENDER_TEXT_CORPUS`, `ONEDIFF_PDF_GATE_DIR`, `ONEDIFF_RENDER_FROM`, `_COUNT`, `_EVERY`, `_JOBS`, `_OUT`); `tools/run-render-gate.sh` runs them in queued chunks.  # Scans and images How `RecursiveAI.OneDiff.Ocr` reads a scanned PDF or an image into the compare model, what the possible-OCR-error flag means, and how the scan gold set is built and scored. ## 1. What is read as a scan `OcrDocumentSource` reads PNG, JPEG and TIFF images (recognized by their signatures, never by file name) and PDFs whose pages carry no text of their own. - A PDF is a scan when none of the pages examined has at least `MinTextLayerCharacters` (default 20) non-space characters in its text layer. All pages are examined up to 8; beyond that, 8 pages spread evenly. A PDF none of whose pages has a content stream is not a scan but a damaged file, and is left to the PDF source, which refuses it. When the pages render but the engine recognizes no word on any of them, the document carries the warning `ocr.nothingRecognized` (the diagnostic `source.<side>.ocr.nothingRecognized`), and strict settings throw on it. - A PDF with an invisible recognized-text layer therefore counts as a digital PDF and is read by the PDF source. - An unreadable or encrypted PDF is not a scan; the PDF source reports why. - Each image frame is one page. An image is wrapped in a one-page-per-frame PDF so that markings can be written on it. The marked file is checked against the wrapper's hash (`PdfOrigin.Sha256`), the compare model's `SourceHash` is the image's own hash. `OcrRedline.Register()` registers the PDF source and the scan source, the scan source with a higher priority (`DocumentSources.Register(source, priority)`), so a scanned PDF goes to OCR whichever of `PdfRedline.Register()` and `OcrRedline.Register()` was called last. Without it, the PDF source refuses a PDF that has no text layer and says to reference this package. The `onediff` tool registers it. ## 2. Reading the text The recognizer returns pages of blocks, lines and words, each with a quadrilateral and a confidence. The model is built the way the PDF source builds it from a text layer, with these differences, which exist because two scans of one page are never segmented the same way: - **Block boundaries are soft.** A block that starts right under the open paragraph (a gap of at most 0.9 line heights, in the same column and type size) continues the paragraph, unless the paragraph's last line was a short line that ended a sentence. Where the recognizer cuts running text into blocks is its own choice and differs from scan to scan, so it is not a paragraph break. Inside a block a gap wider than 0.9 line heights (and, with four or more gaps, 1.6 times the median gap) ends a paragraph. - **Clause numbers start list items.** A line that starts with a multi-level number (`2.3`, `4.1.2`) followed by text is a list item with the number as its `list.label` attribute, whatever the spacing, so renumbering reads as a cascade and not as an edit. A clause number that follows a sentence end inside a line also starts one. - **Tables are off by default** (`ReadTables`). The recognizer finds the same table on two scans of a page only some of the time; a table on one side and running text on the other would be a difference of reading. The cell text still reads in order and every difference in it is shown. Set `ReadTables` to get table, row and cell nodes. - **Running furniture.** A line in the top or bottom tenth of a page whose letters resemble (Dice similarity ≥ 0.65 over letter pairs) a line in the same margin of another page is a header or footer. So is a line of digits and punctuation alone, and the words for "page" or "page of" in English, German, French, Spanish, Italian, Dutch, Swedish, Polish and Turkish. One node stands for each running text; the copy read with the highest confidence is the one kept. Page numbers fold to `#`, joined with the characters a scan puts between them (`Page 2/.3`, `Page 2 / 3`); the fold is a ledger entry (`PageNumbers`). - Line breaks and end-of-line hyphenation fold into the ledger as for a PDF (`LineBreaks`, `Hyphenation`). Nothing the recognizer read is corrected or hidden: a recognition error is a change like any other, with the flag below. ### Anchors Every token is anchored by `AnchorKind.OcrWords`: the quadrilaterals of the words (upright page space, points, origin at the lower left of the page as displayed, y up) and the word's confidence. A paragraph's anchor is one box per line. Offsets are UTF-16 offsets into the node's text. ## 3. The possible-OCR-error flag `Change.OcrConfidence` is the lowest confidence over the words the change's edits touch, resolved per edit range on each side (words next to the edit that merely sit in the same unit do not count). A whole inserted or deleted block has no edits; every word of its side counts. `Change.PossibleOcrError` is true when it is below `CompareSettings.OcrConfidenceThreshold`. - The flag never hides, drops or reorders a change. - Moves, formatting and attribute changes have no confidence. - JSON: optional `ocrConfidence` and `possibleOcrError` per change, `settings.ocrConfidenceThreshold` when not the default. Schema version stays 1. - **Default threshold 0.95**, chosen on the dev scan set only (see §5): a lower threshold has higher precision and lower recall. | threshold | precision | recall | |---|---|---| | 0.80 | .946 | .408 | | 0.90 | .939 | .484 | | 0.95 | .898 | .588 | | 0.97 | .877 | .676 | | 0.98 | .869 | .717 | | 0.99 | .851 | .835 | | 0.995 | .844 | .872 | Precision is the share of flagged changes that no real edit explains; recall is the share of such changes that are flagged. ## 4. Gold set method (scan-v1) `tools/OneDiff.GoldSets.Scan build` takes the PDFs of the digital gold set and scans each file of a pair independently: it renders at 200 or 300 dpi, blurs (sigma 0, 0, 0.7 or 1.2 at 300 dpi), skews by up to ±2°, adds salt-and-pepper noise (density 0, 0, 0.002, 0.006) and encodes each page as PNG or JPEG (quality 90, 75 or 55). All choices come from a seeded generator keyed by the file's name and the seed (20261104), so the set is reproducible. Languages: the Latin-script ones (en, de, fr, es), because the bundled recognizer reads Latin, Greek and Cyrillic. The labels are the digital set's own. A scan's nodes are matched to the gold blocks by similarity; positions are carried over by a minimal character alignment of the whole page's text against the gold text, so a gold block that the recognizer cut into several nodes, or several that it joined, still receive their changes. The scorer's noise rows count a **spurious change**: a text change that overlaps no gold unit, or a move of a block no gold link moves. ## 5. Rows and cost The rows are those of the digital gold sets with the same definitions, plus `noise.changes.share` (spurious changes over all changes), `noise.changes.per10kchars` and `ocr.flag.P/R/share`. The results are in `results/scan-v1-eval.md`. A scan costs the recognizer's time plus the comparison. The recognizer reads about 0.5 s per page on one core of the reference machine; `results/scan-v1-eval.md` has the budget measurement.  # Results The measured results of the 1.0 packages (`RecursiveAI.OneDiff`, `.Documents`, `.Pdf`, `.Ocr` and the `onediff` tool): the text core's exact gates, fuzzing and speed against the managed comparators; the PDF and scan engine's gates and gold-set scores; change semantics and combine; budgets; platforms. Losing rows and gaps are listed, not left out. Nothing is claimed beyond this page. Method: [measurement.md](https://recursiveai.org/onediff/docs/measurement/). Status words: **pass** meets the bar; **FAIL** a gate with failures (listed with its count); **floor** a score recorded as the floor later releases may only raise; **loses** a row where a comparator scores better; **information** measured, no bar. No commercial comparison product is measured or compared against. The adapter packages for Word, spreadsheets, presentations, rendering, OneNLP recognizers and the model ship later; their rows are not part of this page, except the DOCX comparator scores kept at the end as information. ## What was run | | x64 | arm64 | |---|---|---| | CPU | AMD Ryzen 9 7950X3D, 16 cores / 32 threads (avx, avx2, avx512f, avx512bw, bmi2, fma, popcnt, sse4.2) | Apple M2, 8 cores (arm64) | | OS | Linux (x86_64) | macOS (arm64) | | SDK / runtime | .NET SDK 10.0.403, runtime 10.0.14 | .NET SDK 10.0.403, runtime 10.0.14 | | Timed rows | alone on the machine's benchmark queue; load average before the runs 1.6–6.5 (1 min), falling to 1.8–2.5 while they ran (other users' idle processes) | the Mac carries a steady background load of about 2–4 and the benchmark cannot raise its priority: these rows are less quiet than the x64 rows | Code under test: the release candidate plus test-and-tooling changes only (no library code changed). Sister libraries: the rows were measured on the OneFont 1.4.0 and OnePDF 1.18.0 release candidates; the release builds against the published OneFont 1.4.0, OnePDF 1.18.0, OneImage 1.10.5, OneRender 1.3.3 and oneocr 1.0.0, and its test suites passed on those on both machines. Comparators and reference tools run as released binaries and packages: DiffPlex 1.9.0, DiffMatchPatch 5.0.0, git 2.51.2, GNU diffutils 3.12, GNU patch 2.8, the `diff-match-patch` Python package 20241021, W3C Nu Html Checker 26.11.4, BenchmarkDotNet 0.15.8. Data sets (all evaluation splits, never used for tuning): text corpus manifest sha256 `087bd93a…` (9,483 code and prose pairs, 2,599 merge triples; pinned fetch in `tools/corpora-pins.json`); `synth-v1` `e11b5614…` (1,050 pairs, 7 languages); `pdf-v1` `eb10c6b3…` (1,110 pairs rendered to PDF, 60 of them real); `scan-v1` `6fe36af3…` (600 scanned pairs); `combine-v1` `5c55ebb5…` (630 cases; regenerating it from the seed gives the same hash). ## Exact gates (text core) Oracles run as black-box processes. Zero failures in every gate. | Gate | Result | Reference | Verdict | |---|---|---|---| | Myers edit length is the minimum (`diff --minimal` length) | 9,483 / 9,483 | GNU diffutils 3.12 | pass | | Every algorithm's script transforms old into new (Myers, patience, histogram; line, word, character tokens) | 9,483 / 9,483 | — | pass | | Unified patch round trip (make, parse, apply) | 9,483 pairs, 0 failures | — | pass | | dmp patch round trip (make, text, parse, apply) | 9,483 pairs, 0 failures | — | pass | | Our unified output accepted by GNU patch | 9,481 accepted, 2 skipped, 0 failures | GNU patch 2.8 | pass | | Our unified output accepted by `git apply` | 9,481 accepted, 2 skipped, 0 failures | git 2.51.2 | pass | | dmp patch text interop with the Python package, both directions | 9,454 agree, 29 skipped (inputs too large for that package), 0 failures | diff-match-patch 20241021 | pass | | Three-way merge equals `git merge-file -p`, byte for byte, conflict style `merge` | 2,599 / 2,599 | git 2.51.2 | pass | | Same, conflict style `diff3` | 2,599 / 2,599 | git 2.51.2 | pass | | Same, conflict style `zdiff3` | 2,599 / 2,599 | git 2.51.2 | pass | The skipped cases are listed in the gate output; none is a failure. Cases skipped by the oracle (timeouts, over-size) are counted as skips, never passes. ## Fuzzing Seeded mutation runners (`fuzz/OneDiff.Fuzz`, no coverage feedback): inputs are mutations of patches the library writes plus hostile cases; for the change-list reader, mutations of valid comparisons; for the PDF source, mutations of valid PDFs (flips, cuts, splices, repeated chunks, extreme numbers). A crash is an exception other than the documented typed failure, a broken invariant, or an input slower than 3 s. Seed 1, release build, one process per target, all seven run side by side on the x64 machine (8 cores reserved; load average 11–16). | Target | Duration | Executions | Crashes | |---|---|---|---| | unified-parse | 600 s | 368,289,410 | 0 | | unified-apply | 600 s | 320,255,298 | 0 | | dmp-parse | 600 s | 400,521,790 | 0 | | dmp-apply | 600 s | 358,334,639 | 0 | | unified-roundtrip | 600 s | 82,887,489 | 0 | | dmp-roundtrip | 600 s | 10,275,281 | 0 | | comparison-json (change-list reader) | 600 s | 43,825,170 | 0 (slowest input 23.4 ms) | | pdf-source (new for this release: damaged and image-only PDFs are refused) | 60 s smoke | 23,310 (13,362 read, 9,948 refused with the typed failure) | 0 | ## Document exact gates Every pair is compared with the Legal and the Strict settings. | Gate | Result | Verdict | |---|---|---| | `synth-v1` eval: Verify passes, fallback level 0 | 1,050 / 1,050 pairs, fallback 0 | pass | | `pdf-v1`: Verify, fallback 0, `Check()` clean | 2,220 comparisons, 0 failures | pass | | `pdf-v1`: marks on the new document re-extracted with OnePDF equal the change list | 2,220 comparisons, 0 failures | pass | | `pdf-v1`: marks on the old document, re-extracted likewise | 2,220 comparisons, 0 failures | pass | | `pdf-v1`: a second run gives byte-identical comparison JSON and marked PDFs | 2,220 comparisons, 0 failures | pass | | `scan-v1`: Verify, fallback 0 | 600 pairs × Legal/Strict, 0 failures | pass | | `scan-v1`: flags and confidences round-trip through JSON | 32,646 changes (17,777 flagged), 0 failures | pass | | `scan-v1`: marks on the new and old pages re-read | 54,334 marks, 0 failures | pass | | `scan-v1`: determinism (two runs, same bytes) | 100 pairs, 0 failures | pass | | Rendered redline PDF (OneRender 1.3.3) read back equals the change list, inline and side by side each: text corpus | 638 pairs, 0 failures | pass | | The same, text-corpus pairs over 300 kB | 24 pairs, 0 failures | pass | | The same, `pdf-v1` | 1,110 pairs, 0 failures (the first run of the release gate failed 85 pairs, inline layout only; fixed, see below) | pass | | HTML redlines parsed back: every mark carries its change, text equals the change list | 96,000 generated pages (12,000 comparisons × 8 styles and layouts) and 1,324 corpus pages, 0 mismatches | pass | | HTML pages against the W3C Nu Html Checker 26.11.4 | 370 pages, 0 errors, 0 warnings | pass | | Seeded invertibility sweep (random edit sequences) | 60,000 cases, 0 failures | pass | | Seeded alignment sweep (labelled and table pairs) | 60,000 cases, 0 failures | pass | | Seeded grouping sweep (every preset) | 60,000 cases, 0 failures | pass | | Seeded semantics sweep (random contracts: complete, groups partition the changes) | 60,000 cases, 0 failures | pass | | Combine sweep, 1 to 4 reviewers | 60,000 cases, 0 failures | pass | | Chain sweep, 3 to 5 steps | 20,000 cases, 0 failures | pass | **The rendered `pdf-v1` failure, fixed.** A table cell whose text wraps over two lines in the source PDF (for example "6" and "months") kept the line break in its node text; the inline page drew the cell as a stack of short lines, and the re-extraction read the neighbouring cell's text between the cell's lines. The grid reading now joins the lines of a cell with a space (a word broken by a hyphen keeps its break so the hyphenation fold still joins it). The smallest failing pair is a test (`RenderedReadbackTests`). Re-run on the fixed code: `pdf-v1` Verify, fallback 0, `Check`, marks re-read and determinism on 1,110 pairs × Legal/Strict, the rendered read-back inline and side by side on 1,110 pairs, and the text corpus (638 pairs and 24 over 300 kB): 0 failures. The PDF digest slice (20 pairs) is identical on x64 and arm64. **After the cell-text pairing rule** (a paired table cell with one unmatched text block on each side pairs them), the rows it can touch were scored again on the final code: `synth-v1` eval (above: edited, unrelated and changes per gold unit improved, every other row identical), change semantics (every row identical), `combine-v1` (every row identical, digest unchanged), `scan-v1` (every row identical, 600 pairs), the `scan-v1` exact gates on 100 pairs (every 6th; 5,322 changes, 8,921 marks, 20 determinism re-checks, 0 failures), and the alignment, invertibility and grouping sweeps at 60,000 cases each (seeds 1–60,000 in two chunks), 0 failures. ### Same bytes on both machines A fixed slice of `pdf-v1` (20 pairs, every 56th) and of `scan-v1` (20 pairs, every 30th), plus generated and corpus inputs. Each file below has the same SHA-256 on x64 and arm64. | Digest | SHA-256 | Equal | |---|---|---| | `pdf-v1` slice: inputs, comparison JSON, marked PDFs (new and old) | `e8c27c0e…` | yes | | `pdf-v1` slice: rendered redline PDFs, inline and side by side | `f877ec66…` | yes | | Rendered text corpus (every 37th pair), PDFs | `21f32088…` | yes | | Generated inputs, comparison JSON and marked PDFs (10 seeds) | `b2a02d49…` | yes | | `scan-v1` slice: JSON and marked PDFs (OCR source) | `1a421327…` | yes | | HTML pages (random documents in every style and layout, corpus pairs) | `2f32fe58…` | yes | | `combine-v1` first 20 cases: combination JSON and report pages, two runs each | `2c58a823…` | yes | | `synth-v1` slice: comparison JSON of 20 evaluation pairs (4 documents × 5 variants: en, de, zh, ar) | `ab579d71…` | yes (also equal to the digest before the cell-text pairing rule) | ## Gold sets Paired bootstrap over documents, 10,000 resamples, seed 20261104, 95% percentile intervals. Every row is scored once on the evaluation split. All rows equal or beat their floors except one, marked. ### Alignment, boundaries, fragmentation (`synth-v1` eval) Gold documents as input; 210 documents, 1,050 pairs. | Row | Score [95% CI] | Floor | Verdict | |---|---|---|---| | align F1, same | 0.9999 [0.9998, 1.0000] | 0.9999 | floor held | | align F1, edited | 0.9881 [0.9834, 0.9924] | 0.979 | above floor (was 0.9793) | | align F1, moved | 0.9880 [0.9749, 0.9979] | 0.988 | floor held | | align F1, split | 0.9960 [0.9862, 1.0000] | 0.996 | floor held | | align F1, merged | 1.0000 [1.0000, 1.0000] | 1.000 | floor held | | align F1, unrelated | 0.9552 [0.9360, 0.9717] | 0.928 | above floor (was 0.9278) | | boundary F1 | 0.9144 [0.9036, 0.9246] | 0.914 | floor held | | boundary precision / recall | 0.9135 / 0.9152 | .912 / .911 | floor held | | fragmentation: changes per changed sentence | 1.0047 [1.0028, 1.0070] | 1.005 | floor held | | fragmentation: changes per gold unit | 1.1553 [1.1286, 1.1847] | 1.171 | better than floor (a lower count is better; was 1.1712) | | display regions per changed sentence / per unit | 1.0047 / 1.0164 | — | information | | missed changed sentences | 0.0018 [0.0003, 0.0038] | 0.0018 | floor held | | Verify passes / fallback rate | 1.0 / 0 | 1.0 / 0 | pass | Fragmentation counts changes, not display regions, and is read beside boundary precision (definition in [measurement.md](https://recursiveai.org/onediff/docs/measurement/#scoring)). These are gold-set rows with floors, not comparator rows. ### Change semantics (`synth-v1` eval) | Row | Score [95% CI] | Floor | Verdict | |---|---|---|---| | type accuracy, all typed units | .9986 [.9971, .9997] | .9986 | floor held | | resolved-value accuracy, all typed units | .9983 [.9966, .9997] | .9983 | floor held | | amount type / value | 1.0 / .9965 ([.9887, 1]) | same | floor held | | date, duration, percentage, defined term, number (type and value) | 1.0 | 1.0 | floor held | | reference | .9869 [.9727, .9970] | .9869 | floor held | | cascade group precision / recall | .9778 [.9600, .9933] / .9809 [.9642, .9940] | .978 / .981 | floor held | | rename group precision / recall | 1.0 / 1.0 | 1.0 / 1.0 | floor held | | substantive filter precision / recall | .9986 [.9971, .9997] / .9989 [.9971, 1.0] | .9986 / .9989 | floor held | Known limit: the remaining cascade misses are Japanese reference edits merged with a neighbouring adverb change into one unit. ### Combine and chains (`combine-v1`) 630 cases from the 210 evaluation documents with 2, 3 and 4 reviewers. | Row | Score [95% CI] | Floor | Verdict | |---|---|---|---| | invertible per reviewer, check passes, fallback 0 | 630 / 630 | 630 / 630 | pass | | attribution accuracy | .9937 [.9910, .9962] | .9937 | floor held | | shared-edit merge precision / recall | .9787 [.9610, .9927] / .9956 [.9886, 1.0000] | .9787 / .9956 | floor held | | conflict precision / recall | .9648 [.9453, .9819] / .9843 [.9723, .9953] | .9648 / .9843 | floor held | ### Reading a PDF (`pdf-v1`, every fifth pair) The PDF source run on 210 synthetic pairs (222 compared, 0 failures), its nodes mapped onto the gold blocks (20,689 of 22,191 content nodes matched a gold block; the rest are page furniture and extraction artefacts, which count as unrelated). | Row | Score [95% CI] | Floor | Verdict | |---|---|---|---| | align F1, edited | .9361 [.9171, .9542] | .909 | above floor | | align F1, moved | .8429 [.7460, .9241] | .843 | equal at three decimals (the interval's centre is unchanged; moved precision .756, recall .952) | | align F1, split | 1.0000 | .781 | above floor | | align F1, merged | .9706 [.9206, 1.0000] | .846 | above floor | | align F1, unrelated | .5386 [.4756, .5966] | .263 | above floor (low because page furniture has no gold block) | | boundary F1 | .8297 [.7994, .8580] | .792 | above floor | | fragmentation: changes per changed sentence | 1.0057 [1.0012, 1.0109] | 1.006 | floor held | | fragmentation: changes per gold unit | 1.1735 [1.1325, 1.2185] | 1.174 | floor set (re-based from 1.135; a lower count is better): the earlier value counted wrong matches of short equal cells across table rows. Since grids are read as tables, an inserted or deleted row costs one change per cell, while boundary F1 and alignment rose. One change per row is planned | | missed changed sentences | .0423 [.0269, .0598] | .108 | above floor | | Verify passes / fallback | 1.0 / 0 | 1.0 / 0 | pass | ### Scanned pairs (`scan-v1`) 120 evaluation documents (30 per language: en, de, fr, es), 600 pairs. "Digital" is the same pairs read from the digital PDFs by the PDF source, for comparison; the digital column is information. | Row | Scan [95% CI] | Floor | Digital | |---|---|---|---| | align F1 same | .7041 [.6949, .7131] | .704 | .9569 | | align F1 edited | .4997 [.4700, .5281] | .500 | .9289 | | align F1 moved | .3205 [.2582, .3814] | .321 | .8443 | | align F1 split / merged | .0552 / .1085 | .055 / — | .9867 / .9905 | | align F1 unrelated | .0524 [.0430, .0625] | .052 | .4658 | | boundary F1 | .2367 [.2155, .2589] | .237 | .8508 [.8300, .8705] | | boundary precision / recall | .1400 / .7671 | .140 / .767 | .7921 / .9190 | | frag.sentence / frag.unit | 1.0464 / 1.1115 | 1.046 / 1.112 | 1.0067 / 1.1802 | | missed changed sentences | .0945 [.0762, .1139] | .095 | .0100 | | noise.changes.share | .8649 [.8503, .8782] | .865 | .4314 | | ocr.flag.P (flagged changes no real edit explains) | .9320 [.9224, .9406] | .932 | — | | ocr.flag.R (such changes that are flagged) | .6197 [.5889, .6492] | .620 | — | | Verify passes / fallback | 1.0 / 0 | 1.0 / 0 | 1.0 / 0 | The scan boundary and noise rows are low because recognition errors make most changes "noise" (86%), a limit of the recognition engine; the flags mark such changes (precision .93, recall .62 at threshold .95). The digital column is lower than at the previous measurement (boundary F1 .851 against .879, edited .929 against .944, unrelated .466 against .438; intervals overlap); it is the PDF source reading the same pairs after later changes to clause numbers and tables, and it is not a floor row. ## Speed: text core against the managed comparators Comparators: DiffPlex 1.9.0 and DiffMatchPatch 5.0.0 (the diff-match-patch authors' C# package), restored in the benchmark project only. Ratio = ours ÷ comparator, so lower is better. Bar: geometric mean per comparator and area at most 1.0, no row above 1.25. **Settings.** Single-threaded, workstation non-concurrent GC; DiffMatchPatch `Diff_Timeout = 0`; no cleanup in the diff rows; DiffPlex `ignoreWhitespace: false` with an explicit word separator set; DiffMatchPatch's line mode maps lines to characters inside the timed work (its helper is not public); cleanup rows time the copy of the input diff on both sides. BenchmarkDotNet 0.15.8, one launch per row with its default warm-up and iteration counts, `Release`. Inputs are corpus pairs of about 2 KB (Small), 40 KB (Medium) and 300–400 KB (Large; lines and words only), named in the raw tables. Each area ran as its own group of under 20 minutes. Raw BenchmarkDotNet output, the machine record and the input and edit-distance listing are kept with the run; peak working set is measured once per row in a cold child process. | Area | Comparator | x64 geomean | x64 worst row | arm64 geomean | arm64 worst row | Verdict | |---|---|---|---|---|---|---| | lines (3 algorithms × 3 sizes) | DiffPlex lines | 0.19 | 0.32 | 0.19 | 0.34 | pass | | lines | diff-match-patch line mode | 0.52 | 0.72 | 0.54 | 0.78 | pass | | words (3 × 3) | DiffPlex words | 0.29 | 0.81 | 0.35 | 0.93 | pass | | chars (3 × 2) | DiffPlex characters | 0.01 | 0.08 | 0.01 | 0.08 | pass | | chars | diff-match-patch `diff_main` | 0.22 | 0.56 | 0.24 | 0.67 | pass | | cleanup, semantic (2 sizes) | diff-match-patch | 0.32 | 0.48 | 0.33 | 0.48 | pass | | cleanup, efficiency (2 sizes) | diff-match-patch | 0.51 | 0.89 | 0.53 | 0.93 | pass | | patch apply (2 sizes) | diff-match-patch | 0.61 | 0.65 | 0.61 | 0.64 | pass | | patch make (2 sizes) | diff-match-patch | 0.85 | **1.13** | 0.83 | **1.02** | pass (geomean under 1.0; worst row under 1.25) | | patch toText | diff-match-patch | 0.21 | 0.32 | 0.18 | 0.26 | pass | | patch fromText | diff-match-patch | 0.09 | 0.11 | 0.07 | 0.08 | pass | Three-way merge has no managed comparator and is covered by the exact gate above, not a speed row. - **Words are not equal work.** DiffPlex's word differ cuts words at the separators it is given, so it does not see our token stream: it reports fewer edits on the same input (Small 48 against our 57–61, Medium 60 against 69, Large 840 against 1,114). The ratios compare the time to produce each library's own script. Lines and characters agree on edit distance between all implementations except where a heuristic algorithm (patience, histogram) finds a longer, equally valid script than Myers. - **Not covered.** No word rows against diff-match-patch (it has no word mode), no large-size character rows, no large-size cleanup or patch rows, no multi-threaded rows (the text core is single-threaded). Native tools (git, GNU diff) are not compared. ### Allocations (bytes per call, x64; arm64 agrees within 1%) | Row | Ours | Comparator | Ratio | |---|---|---|---| | lines, Medium, Myers | 105,960 | DiffPlex 317,592 / diff-match-patch 342,648 | 0.33 / 0.31 | | lines, Large, Myers | 960,825 | DiffPlex 2,132,880 / diff-match-patch 2,880,414 | 0.45 / 0.33 | | words, Large, Myers | 3,031,091 | DiffPlex 10,878,894 | 0.28 | | chars, Medium, Myers | 1,960 | DiffPlex 3,637,006 / diff-match-patch 229,568 | 0.00 / 0.01 | | semantic cleanup, Small | 1,216 | diff-match-patch 21,512 | 0.06 | | efficiency cleanup, Medium | 0 | diff-match-patch 328 | 0.00 | | patch make, Medium | 253,104 | diff-match-patch 392,096 | 0.65 | | patch apply, Medium | 325,328 | diff-match-patch 487,464 | 0.67 | Every measured row allocates less than its comparator (allocation ratio at most 0.67). ### Memory Peak working set above the inputs, x64, cold child process: | Row | Ours | DiffPlex | diff-match-patch | |---|---|---|---| | lines, Large (Myers) | 5.7 MB | 5.3 MB | 4.0 MB | | words, Large (Myers) | 14.7 MB | 12.7 MB | — | | chars, Medium (Myers) | 2.3 MB | 7.5 MB | 2.8 MB | At the two largest rows where we are higher the process baseline dominates (within about 8–16% of DiffPlex's); no bar is set on peak memory for the text core. **Linear growth** (lines, Myers, bytes allocated per diff on inputs of 2 KB, 40 KB and 317 KB): 9,144, 230,184 and 1,514,568, that is 4.6, 5.8 and 4.8 bytes per input byte. The time grows the same way (1.7 µs, 68 µs and 792 µs). ## Budgets A 100-page pair with about 300 changes, 4 cores (`DOTNET_PROCESSOR_COUNT=4`), one untimed warm-up and three timed repetitions per part, median reported, each part in its own process; method in [measurement.md](https://recursiveai.org/onediff/docs/measurement/). The peak is the operating system's maximum resident set size. Budgets are targets of this release. | Row | Budget | x64 | arm64 | Verdict | |---|---|---|---|---| | PDF pair: read both, compare, mark the new pages | 2 s | 841 ms, peak 478 MB | 726 ms, peak 485 MB | pass | | The same pair to a rendered redline, inline | 2 s and peak 1 GB | 1,178 ms, peak 639 MB | 1,197 ms, peak 608 MB | pass | | The same, side by side | information | 1,717 ms, peak 718 MB | 1,738 ms, peak 686 MB | — | | Memory above the inputs per page: markup / rendered inline / side by side | information | 4.1 / 5.7 / 6.5 MB | 4.1 / 5.3 / 6.1 MB | — | | Scan pair: compare + mark after OCR (20-pair slice of `scan-v1`, about 6 pages per pair) | OCR time + 1 s | at most 145 ms (median 6 ms); OCR 1.7 s per pair (332 ms per page) | at most 122 ms (median 5 ms); OCR 9.2 s per pair (1.7 s per page) | pass | The scan OCR time is the recognition engine's, not ours; on the Mac it was measured on a loaded machine. ## Platforms | Check | x64 | arm64 | Verdict | |---|---|---|---| | Whole-solution tests (all test projects, Release) | 864 tests: 848 passed, 16 skipped, 0 failed | 864 tests: 846 passed, 18 skipped, 0 failed | pass | | Build warnings with warnings as errors | 0 | 0 | pass | | NativeAOT smoke app (every package's main path), `-warnaserror` | 0 warnings; runs and prints `ok`; 55.2 MB | 0 warnings; runs and prints `ok`; 53.8 MB | pass | | NativeAOT `onediff` tool, `-warnaserror` | 0 warnings; `diff` exits 1 and prints the change; 48.3 MB | the same; 47.0 MB | pass | | Pack dry run: exactly `RecursiveAI.OneDiff`, `.Documents`, `.Ocr`, `.Pdf`, `.Tool`, each with the licences, notices and readme | 5 packages, set matches | 5 packages, set matches | pass | The test run and the pack dry run were repeated after the adapter packages moved to the renamed document library: the same counts (864 tests, 0 failed; 5 packages) on both machines. Skipped tests are those that need a native host or a file set that is not present (including the pinned failing read-back test above). Browser WebAssembly (headless Chrome 155, x64): Core 361 tests, 353 passed, 0 failed, 8 skipped; `.Documents` 200 (197 passed, 3 skipped); `.Pdf` 135 (119 passed, 16 skipped); `.Ocr` 18 (8 passed, 10 skipped); skips are tests that need files, threads, processes or AES, marked native. The `.Ocr` skips include every test that reads a scan: under WebAssembly the image library (OneImage 1.10.5) throws a `TypeLoadException` while rasterizing, so scans are not supported in the browser until a OneImage fix (expected in OneImage 1.11). ## DOCX comparators on docx-v1 (information; not part of this release) Set: `docx-v1`, sha256 `62b43875b9746cf41e59fafefa643e3ddf02c3301c490a6cf1eae7ce6697f91e`: the 1,050 evaluation pairs of the synthetic set rendered to Word files in six looks. The free comparison tools were run as released binaries through their public interfaces, one run each, with default settings except where noted in [`results/docx-v1-comparators.md`](https://github.com/recursive-ai-oss/onediff-csharp/blob/6443915c4136cc8c5a49fa2543322011665f309e/docs/results/docx-v1-comparators.md) (Docxodus 15.0.0 package, OfficeIMO.Word 3.5.3, WmlComparer through python-redlines 2.0.0, LibreOffice 26.8.2.2 headless). **Indicative**: one run each, on a shared machine. Bootstrap 95% intervals over documents (1,000 resamples). **OneDiff's own rows on this set: pending: the DOCX package is not in this release.** The engine's tracked-changes DOCX output is not scored here. | Tool | Pairs run | Accept-all = new | Reject-all = old | |---|---|---|---| | Docxodus | 1,050 / 1,050 | 1,050 | 1,050 | | OfficeIMO | 1,050 / 1,050 | 1,050 | 958 | | WmlComparer | 1,050 / 1,050 | 1,050 | 1,050 | | LibreOffice | 884 / 1,050 (166 files of one look did not load) | 883 | 763 | | OneDiff | pending: the DOCX package is not in this release | pending | pending | | Row | Docxodus | OfficeIMO | WmlComparer | LibreOffice | OneDiff | |---|---|---|---|---|---| | align F1, edited | .952 [.943, .960] | .934 [.923, .944] | .928 [.916, .939] | .802 [.771, .829] | pending | | align F1, moved | .919 [.885, .948] | 0 | 0 | 0 | pending | | align F1, split | .992 [.972, 1.000] | 0 | 0 | 0 | pending | | align F1, merged | .989 [.970, 1.000] | 0 | 0 | 0 | pending | | align F1, unrelated | .851 [.820, .878] | .731 [.694, .766] | .740 [.703, .776] | .094 [.082, .106] | pending | | boundary F1 | .689 [.668, .710] | .337 [.312, .364] | .561 [.526, .594] | .435 [.408, .462] | pending | | fragmentation: changes per changed sentence | 1.144 [1.115, 1.177] | 1.000 | 1.060 [1.048, 1.073] | 1.050 [1.038, 1.064] | pending | | fragmentation: changes per gold unit | 1.405 [1.362, 1.452] | 1.215 [1.184, 1.249] | 1.314 [1.278, 1.351] | 1.350 [1.308, 1.396] | pending | | missed changed sentences | .072 [.063, .082] | .050 [.041, .059] | .043 [.035, .051] | .221 [.182, .261] | pending | The comparators' zeros for moved, split and merged are not misreads: only Docxodus writes move markers, and none of the others can express a link between paragraphs. OfficeIMO's fragmentation of 1.000 is a paragraph per change (its boundary F1 is .337). Speed of the comparators on joined pairs of about 5, 40 and 120 pages is in the linked file (indicative); the DOCX speed rows of OneDiff are pending with the package. Rows where OneDiff's DOCX output and the free tools are compared (strictly better, the ceiling clause, the blind-preference rows) are not published in this release. ## Failures, rows below their floor, and gaps - **PDF source, changes per gold unit: floor re-based** to 1.174 [1.133, 1.219] from 1.135, because the earlier value counted wrong matches of short equal cells across table rows. An inserted or deleted table row now costs one change per cell; making it one change is planned. Every other `pdf-v1` row is at or above its floor. - **At the floor within rounding:** PDF/PDF moved align F1 (.8429 against .843). - **Scan limits:** scan boundary F1 (.237), split and merge recovery (F1 .055 and .109) and moved precision are limited by recognition errors and by paragraphs the recognition engine merges; the flags (precision .93, recall .62) mark rather than hide them. Latin-script scans only. - **Known limit:** one 29.7 MB pair exceeds the renderer's display-list limit; the rendered redline gives a clear error for it. It is a limit, not a failure of the gates above. - **Text-core rows that lose:** none. Patch make on the medium size is slower than diff-match-patch's by 1.13× on x64 and 1.02× on arm64, inside the bar (geomean .85 and .83); peak memory on the largest line and word inputs is above DiffPlex's (5.7 against 5.3 MB and 14.7 against 12.7 MB), and no bar is set on it. - **Not run in this release:** DOCX engine rows and DOCX speed rows against the free DOCX tools, blind-preference rows, grid and slide rows, the model rows, and a WebAssembly timing row; the packages that produce them ship later. There is no managed comparator for PDF, scan, merge or document speed, so those rows are budgets only. - **Platform coverage:** the browser suites ran on x64 only; native AOT, the whole-solution tests and the pack dry run ran on both machines. The scan-time and OCR rows were taken on a 20-pair slice.  # Measuring OneDiff How the exactness gates and the speed rows are reproduced. Everything here is tooling: it lives in `tools/` and `bench/`, nothing in it ships in a package, and the comparators and reference tools it uses are only ever executed or called through their public interfaces. ## Reference tools `tools/install-oracles.sh [ROOT]` installs pinned released binaries under `ROOT/oracles` (default `$ONEDIFF_HOME`, else `~/.cache/onediff`) without touching the system: | Tool | Version | Source | |---|---|---| | git | 2.51.2 | conda-forge binary through a user-local micromamba | | GNU diffutils (diff, diff3) | 3.12 | same | | GNU patch | 2.8 | same | | diff-match-patch (Python) | 20241021 | PyPI wheel (`--only-binary`) | The script prints each tool's own version line and exits non-zero if one differs from its pin. Linux x64 and macOS arm64 are supported. The GPL tools are run as processes by the harness and the benchmarks and are never referenced from a package. ## Corpora `tools/fetch-corpora.py [--root DIR] [repos] [wiki] [synth]` builds the text and merge corpora under `DIR` (default `$ONEDIFF_CORPORA`, else `$ONEDIFF_HOME/corpora`). Nothing is committed. - `text/<source>/<n>/{old,new}`: version pairs from the history of six permissively licensed repositories, from Wikipedia revisions (seven languages) and from a seeded set of edge cases (emoji and joiners, combining marks, CRLF/LF mixes, missing final newline, a 1.2 MB single line, a 150k-line file, empty and identical sides). - `merge/<repo>/<n>/{base,ours,theirs}`: three-way triples from real merges that changed the same file on both sides. - A licence or terms file beside every source, and `manifest.json` with a SHA-256 for every file. `tools/corpora-pins.json` pins every repository to a commit and every Wikipedia page to a list of revision ids, so a fresh fetch yields the same files. The manifest's own SHA-256 is the corpus identity; it is printed at the end of the fetch and recorded in every gate report. ## Exactness gates `tools/OneDiff.Oracles` is a console tool that runs the reference tools as black boxes over the corpus: ``` dotnet run -c Release --project tools/OneDiff.Oracles -- run \ --corpus ~/.cache/onediff/corpora --oracles ~/.cache/onediff/oracles --out out --cores 8 ``` | Gate | Checks | |---|---| | `min-edit` | our Myers edit distance on lines equals `diff --minimal`'s (lines end at `\n`, keeping their terminator, as GNU diff cuts them) | | `validity` | for every algorithm and for line, word and character tokens: the script is well formed, equal runs are equal, applying it gives new and undoing it gives old, and no algorithm beats Myers | | `roundtrip-unified`, `roundtrip-dmp` | our patch, applied by our code, reproduces new | | `accept-gnu-patch`, `accept-git-apply` | our unified output applied by GNU patch and by `git apply` reproduces new | | `dmp-interop` | our diff-match-patch patch text applies in the Python package and the package's patch text applies in ours | | `merge-file:merge`, `:diff3`, `:zdiff3` | our three-way merge equals `git merge-file -p` byte for byte in each conflict style, with the conflict count; each mismatch keeps a unified diff in `merge-mismatch/` | The patch and merge gates call the library through one file, `tools/OneDiff.Oracles/Seam.cs`. `--seam oracle` answers from the oracles themselves, so every gate must pass against it; that tests the gate plumbing. Cases run in parallel (`--cores`), are listed in ordinal order, and each finished case is appended to `results-<seam>.jsonl`; rerunning with the same `--out` skips them (`--fresh` starts over). `report.json` has no timings and is byte-identical across runs. Oracle timeouts and inputs too large for the Python package are counted as skips, never as passes. Exit code 1 means some gate has failures. ## Benchmarks `bench/OneDiff.Bench` holds the BenchmarkDotNet suites. The comparators, DiffPlex 1.9.0 and DiffMatchPatch 5.0.0 (the diff-match-patch authors' C# package), are restored for this project only. Inputs are corpus pairs of about 2 KB, 40 KB and 400 KB (the nearest pair available, named in the report), read at run time. | Area | Ours | Comparators | |---|---|---| | lines | `Diff.Lines` per algorithm | DiffPlex line differ; diff-match-patch line mode | | words | `Diff.Words` per algorithm | DiffPlex word differ | | chars | `Diff.Characters` per algorithm | DiffPlex character differ; diff-match-patch `diff_main` | | cleanup | semantic and efficiency | diff-match-patch cleanup | | patch | make, toText, fromText, apply | diff-match-patch patches | Settings, so the work is the same on both sides: - Everything is single-threaded, workstation non-concurrent GC. - No cleanup in the diff rows: each implementation returns its raw edit script. - diff-match-patch runs with `Diff_Timeout = 0` (no time limit), so it finds the full diff. Its line mode maps each distinct line to one character, diffs the characters and reads the result in lines; the package keeps its own line mapping non-public, so the mapping is done in the benchmark, inside the timed work. - DiffPlex's word differ cuts words at the separators it is given (spaces, line ends and common punctuation), so the word rows do not see identical token streams. `info` prints the edit distance each implementation finds on each input; lines and characters agree between DiffPlex and diff-match-patch. - Cleanup rows start each call from a copy of the same diff, and the copy is timed on both sides. `bench/run-bench.sh OUT [BenchmarkDotNet args]` runs everything: the suites (with `ONEDIFF_BENCH_MODE=smoke` for a short indicative pass), the input and edit-distance listing, peak working set per row (one cold run per child process, reported above a load-only baseline), and `results-table.md`: per size, ours against each comparator with time ratio, allocations and allocation ratio, a geometric mean per area and comparator, and the verdict against the bars (geomean at most 1.0, no row above 1.25). `bench/record-run.sh` writes the CPU model, core count, instruction-set flags and the load average before, during and after into the same folder. Timings from a machine that was doing other work are not results. ## Restoring the sister-library previews `RecursiveAI.OneDiff.Pdf` builds on a preview of `RecursiveAI.OnePdf` that is not on nuget.org yet. Until it is, restore needs one extra package source that is kept out of the repository: 1. Put the preview `.nupkg` files (`RecursiveAI.OnePdf`, `.Layout` and `.Fonts` at the version in `Directory.Packages.props`) in a folder and point `ONEDIFF_PREVIEW_FEED` at it. 2. Copy `docs/nuget-local.config.example` to `NuGet.Config` at the repository root. The file is not tracked, so no machine path ever lands in a commit. It maps only the OnePDF package ids to that folder and everything else to nuget.org; the mapping also keeps central package management from warning about several sources. 3. `dotnet restore` as usual. Without the file, restore of the PDF projects fails with an unresolved package version; every other project restores from nuget.org alone. Hosted CI cannot restore the PDF projects until the version is published. ## Gold sets The document engine is measured against labelled pairs. Three kinds of data exist, and each has a fixed role: **synthetic pairs with exact labels** (the scored set), **the same pairs rendered to PDF plus real version pairs** (the PDF gate set and the exact gates), and **real pairs without labels** (invertibility, fallback and re-extraction only). Nothing in this section is committed except the tools; data lives under `~/onediff/eval/` (evaluation) and is regenerated from seeds and pins. ### Synthetic pairs `tools/OneDiff.GoldSets generate --out DIR [--role eval|dev|train] [--seed N] [--docs-per-lang N] [--verify]` writes labelled pairs. - **Documents** are contract-like and written for the generator, in seven languages (English, German, French, Spanish, Chinese, Japanese, Arabic): a title, a preamble, three recitals, a definitions article, five to eight articles of numbered clauses (one to three sentences each), a fee table and a closing line. Sentences come from twelve templates per language with two phrasings each; amounts, durations, dates, percentages, jurisdictions, one defined term and cross-references (`Section 3.2`) are filled in from small value lists. Nothing is copied from any real document. - **Edits** (twenty types) each produce labelled change units: `term-swap`, `value-amount`, `value-date`, `value-duration`, `value-percentage`, `clause-rewrite` (the same content in the other phrasing), `sentence-insert`, `sentence-delete`, `para-insert`, `para-delete`, `move`, `move-edit` (moved and changed), `cascade` (a clause inserted before a referenced one, so labels and cross-references renumber), `term-rename` (the defined term changes everywhere), `split`, `merge`, `format` (bold on a word), `table-cell`, `table-row-insert`, `table-row-delete`. Renumbering that follows from any edit is labelled too (`renumber` units and `cascade-reference` units, group `cascade`). - **Pairs.** Per base document: one mixed pair (`-m`, four to eight edits) and four single-type pairs (`-s1` to `-s4`, one to three edits of one type; the types rotate so each is covered in every language). A pair never edits the same sentence twice, so units never overlap. - **Split.** Per language 60 base documents are shuffled with a seeded generator and cut into 30 `eval`, 6 `dev` and 24 `train` documents. All pairs of a base document share its role. The split does not depend on which role is written: `split.json` lists every document with its role and the SHA-256 of its base text. - **Files.** `pairs/<id>.docs.json` (old and new document as compare-model nodes: kind, parent, text, attributes, formatting runs; a node's id is its position) and `pairs/<id>.labels.json`; `split.json`; `manifest.json` (every file's SHA-256, counts per language and edit type, the SHA-256 of `split.json`); `manifest.sha256`, the **set hash**. The same seed gives the same bytes. - **Labels.** `links`: content blocks (paragraphs and list items) by node id, kind `match`, `move`, `split` or `merge`; `deleted` and `inserted` content blocks; `units`: each human-meaningful change with its `type`, `group` (`rename`, `cascade`), `typed` value (kinds `amount`, `date`, `duration`, `percentage`, `term`, `modal`/`adverb`/`jurisdiction` swaps, `reference`, `number`; values in a language-neutral canonical form such as `USD 12500`, `30 d`, `2027-03-01`) and its character spans on each side; `sentences`: the changed gold sentences. A unit without spans (`split`, `merge`, `renumber`) is structural. - **Sanity gate.** `--verify` runs every pair through the current pipeline with nothing hidden and requires `Verify` to pass at fallback level 0. The generator also refuses a pair in which a difference between the documents is not covered by a unit. ### Keeping evaluation out of training Evaluation data is written only under `~/onediff/eval/`; training data under `~/onediff/train/`, and no tool reads across. `generate --role train` never writes eval pairs, and the roles of a seed are fixed by the shuffle above, so a training set cut from `--role train` of the **same seed** is disjoint from the eval set by construction. `check-split --a EVALDIR --b TRAINDIR` proves it for any two directories: it compares the SHA-256 of every base document and exits 1 on a shared one. Distillation and head training load `train` and `dev` only. Real pairs and the PDF renderings of eval pairs are eval. The eval manifest (set hash) is recorded before any model sees training data. ### The PDF gate set `tools/OneDiff.GoldSets.Pdf render --data DIR [--real DIR] --out PDFDIR` renders every synthetic pair (and every real pair) twice, old and new, with OnePDF's layout engine. The layout is chosen from the pair id (four variants: a single justified, hyphenated column on A5; two columns on A5 with a header; one wide-margin column on A4 with a header; two narrow columns on A5 with a header), all with a "Page n / N" footer, tables, lists, page- and column-spanning paragraphs, soft-hyphenated line ends, CJK (Noto Sans CJK) and Arabic (Noto Sans Arabic, right to left). The file id is derived from the file name, so equal input gives equal bytes. After rendering, the text is extracted again with OnePDF and every content block is looked up in order; `pdf/<id>.blocks.json` keeps node id, label, text and page per block, and the manifest reports how many blocks were located. A block that is not found is a limit of extraction (a title word split across lines, a percent sign in right-to-left text), not a missing label. The gold node ids of a pair are the ids of the source documents; a PDF source numbers its own blocks, so scoring a PDF comparison maps blocks by text through `blocks.json`. ### Real pairs `tools/fetch-real-pairs.py` fetches version pairs into `~/onediff/eval/real-pairs-v1` and records everything in `tools/real-pairs-pins.json` (URLs, repository commits, EDGAR documents, each with the SHA-256 it had; a changed document stops the fetch). Real pairs carry no labels: they exercise invertibility, fallback and re-extraction. Sources and terms are in the script's header; nothing fetched is committed. SEC requests are paced at 4 per second under a declared user agent (`ONEDIFF_SEC_CONTACT`). ## Scoring `tools/OneDiff.GoldSets score --data DIR --out FILE [--pipeline NAME]` runs the reference pipeline with nothing hidden (`CompareSettings.Strict`, no throwing), or reads `Comparison` JSON from `--cmp DIR` (`<id>.cmp.json`), checks every file against the manifest, scores each pair and writes rows. A row is a metric on a scope (`all`, `lang:xx`, `mixed`, `edit:<type>`) with its value, a bootstrap 95% interval, the number of documents and pairs, the set hash and the pipeline name. - **Content blocks** are paragraphs and list items; table, row, cell and section nodes carry no text and are not scored. - **Alignment F1 per link type.** A link is typed from its alignment kind and texts: `same` (a match with equal text), `edited` (a match with different text), `moved`, `split`, `merged`; a content block with no link is `unrelated` (deleted or inserted). A predicted link is correct when its type and its sets of old and new node ids equal a gold link's. Precision, recall and F1 per type, from counts summed over documents. - **Boundary F1.** A change unit's spans are trimmed of white space; the boundaries are the (side, node, offset) of every span start and end. Predicted spans come from the engine's changes of all kinds except attribute, split and merge (those have no text span); a side of a change is mapped to its nodes' text. Boundary precision, recall and F1 are over exact matches. - **Fragmentation.** Two counts are kept apart, because a display region can be as large as one likes and still look like "one per sentence". - A change's **text extent** is the union of its sides' trimmed spans. A move change spans its whole node, so it has an extent only for a gold move unit; the edits inside a moved node are changes of their own. Attribute, split and merge changes have none. - A **changed gold sentence** is a sentence of the new document that carries a text change (of the old document, for deleted text). A gold unit's target is its spans. - `frag.sentence` is the mean, over changed sentences touched by at least one change, of the number of **changes** whose extent overlaps the sentence; `frag.unit` is the same per gold unit (the changes overlapping any of its spans; a gold move unit counts move changes). 1.0 means one change per sentence or unit; more means a change is shown in pieces. This is the fragmentation row. - `regions.sentence` and `regions.unit` are the same means over **display regions** (change groups) that hold a counted change. They say how the pieces are presented, never how many there are: a whole-paragraph region holding five changes is five changes and one region. Read `frag` for unit quality and `regions` for display. - `missed.sentence` is the share of changed sentences no change touches. - Fragmentation is meaningful only beside boundary precision: one unit spanning a whole paragraph also has one change per sentence, and pays in `boundary.P`. No row is read alone. - **Pairs with no difference.** A pair whose two documents are identical (the generator can undo a defined-term rename with a second one) has nothing to compare, whatever its unit labels say. Its unit and sentence labels are left out of the boundary and fragmentation counts and `pairs.nodiff` reports how many pairs were left out. Alignment rows still count them. - **Aggregation.** Counts are summed per base document (all pairs of the document in the scope); documents are the resampling unit. The interval is the percentile bootstrap (2.5% and 97.5%) of the metric recomputed on 10,000 resamples of the documents with replacement, seed 20261104. The tool also prints the per-document distribution (minimum, median, 90th percentile, maximum) of boundary F1, changes per changed sentence and regions per changed sentence. `--resamples N` changes the resample count (0 skips the intervals, for sweeps that compare point values); every reported row uses 10,000. `<out>.missed.txt` lists, per pair, the changed sentences no change touched. A metric whose denominator is zero in more than 10% of resamples has no interval. - **Verification** is scored too: `verify.pass` and `verify.fallback` are the shares of pairs that passed and that needed a coarser grouping. ### Semantic rows Scored on the engine's own comparisons of the gold documents (`CompareSettings.Strict` with `CollapseCascades` and `TypedChanges` on, no throwing; the gold nodes are the engine's nodes), so they are not computed for PDF or DOCX renderings. Definitions (docs/semantics.md has the rules being measured): - **Mapping.** A gold unit is mapped to the change whose text extent overlaps its old spans most, and to the change that does for its new spans (one change for a replacement; a deleted and an inserted table cell are two). Moves map only to move changes and nothing else maps to them. A `renumber` unit has no spans; it maps to the attribute change that moved that list label (from and to equal, ignoring a closing full stop). - **`type.acc.<kind>`.** Over gold units with a typed value of the kind that were mapped: the share whose mapped change(s) have the engine's type for it. Kinds and types: `amount` Amount, `date` Date, `duration` Duration, `percentage` Percentage, `term` DefinedTerm, `reference` Reference, `number` (the label of a `renumber`) Numbering. `modal`, `adverb` and `jurisdiction` swaps are plain substantive text changes with no type and are not scored. `type.acc.all` sums the kinds; `type.coverage` is the share of typed units (those kinds) that were mapped to any change. - **`value.acc.<kind>`.** Over the same mapped units: the share whose old value from the old-side change and new value from the new-side change equal the gold value exactly. Gold values are canonical strings (`USD 25000`, `30 d`, `2027-03-01`, `Software`); a gold reference is the text of the reference (`Section 8.4`) and is compared by its number, a gold label by itself without a closing full stop. - **`cascade.P` / `cascade.R`, `rename.P` / `rename.R`.** A gold group is the set of units with that `group` in a pair. One of the engine's groups of that kind *finds* it when the units that its changes are mapped to are exactly the gold set. Precision is found groups over the engine's groups of the kind, recall found groups over gold groups. - **`substantive.P` / `substantive.R`.** Gold substantive is every unit type except `renumber`, `cascade-reference`, `format`, `split` and `merge` (list renumbering and its references, formatting and paragraph structure; split and merge have no spans and are not scored). A unit is predicted substantive when a change mapped to it is. A unit that maps to no change counts as not predicted. - **Pairs with two renames.** In some pairs the generator renamed the term twice and the second rename overwrote the pieces of the first; the first rename's units then describe text that is not in the documents. Their rename units are left out of every semantic row (`pairs.renamechain` counts the pairs: 6 of 210 on dev). This is a label artefact; synth-v2 does not make it. - Pairs with identical documents are excluded as for the other rows. The bootstrap is the same (documents, 10,000 resamples, seed 20261104). - `<out>.semantic.txt` lists every miss with the pair id. ## DOCX gold set and comparison tools The DOCX set is the evaluation split of the synthetic pairs written as Word files, so the labels carry over exactly. Everything here is indicative until a final run is published (`docs/results/docx-v1-comparators.md`). ### Rendering and block maps `tools/OneDiff.GoldSets.Docx render --data DIR --out DIR` writes `docx/<id>.{old,new}.docx`, `<id>.blocks.json` and a manifest whose hash is the set identity. Packages are assembled from OOXML parts and opened and read back through OneDocument, which is also how list labels are checked (`labelsOk` equals `labelsTotal` on every side). Six looks, picked by a hash of the pair id, so each language and edit type appears in each: three plain looks (different fonts, page set-up and table styles), one with bookmarks and cross-reference fields, one with a footnote and a comment on the title, one with a content control around the title. Every look has numbering, a table with a header row and a merged-cell signature box, a header with a picture and a footer with page-number fields. A block map lists, per side, the gold content blocks in order (`node`, `kind`, `label`, `text`, and for PDFs the page). PDF and DOCX renderings write the same shape. The scorer maps a source's content nodes onto gold blocks by the longest common subsequence of equal texts after dropping white space, soft hyphens and direction marks, so order is kept and repeated texts pair up in order; a block's list label may prefix the text (a PDF prints it). Offsets are mapped by counting the characters kept. A source node that matches nothing gets an id past the gold nodes, so lookups never fail and the node counts as unmatched in the printed `block map: matched/total` line. - `score --cmp DIR --blocks DIR` maps comparisons made from rendered files (read as `<id>.cmp.json`); `OneDiff.GoldSets.Pdf compare --pdf DIR --out DIR [--every N]` makes them from PDFs. - `score --changes DIR --blocks DIR` maps another tool's output, read as `<id>.changes.json` (schema `onediff.changes/1`: two line views and the changes as line intervals). A missing file scores as an empty result and is counted as "pairs without output". `--ids FILE` limits the pairs. ### Tracked-changes outputs `OneDiff.GoldSets.Docx read --tool T --tracked DIR --docx DIR --out DIR` reads each tool's DOCX through OneDocument's markup and revision model, writes `<id>.changes.json`, and runs accept-all and reject-all on copies, comparing the paragraph sequence with the new and old input (`roundtrip.jsonl`). The two views are the document with every revision rejected and with every revision accepted, one line per non-empty paragraph; a removed paragraph mark joins a paragraph to the next (ECMA-376 §17.13.5). A change is a run of adjacent revisions in one paragraph with no non-blank unchanged text between them (white space between two revisions does not split a change); the two halves of a move are one change, paired by the name on their range markers; formatting revisions are not changes. Every change is its own display region, so `regions.*` equal `frag.*` for tool outputs. One package quirk is repaired before reading: some tools type every `.xml` part through one default content type and give the main part no override; the reader adds the override and changes nothing else. ### Comparison tools Run only as released binaries, from the harness in `tools/OneDiff.Comparators` (in-process .NET packages), `tools/comparators/wmlcomparer.py` (a Python package that carries the engine as a compiled binary; install with `--only-binary` into a virtual environment) and `tools/comparators/libreoffice_compare.py` (headless office through its UNO bridge, run with the Python shipped in the office's program folder). `tools/install-libreoffice.sh` installs LibreOffice 26.8.2 from the release tarball (sha256 checked) under the oracle root, Linux x64 or macOS arm64; the office is only executed. Pins and settings are in `docs/results/docx-v1-comparators.md`. Each runner records one line per pair (`runs.jsonl`: ok, milliseconds, output size, error) and skips recorded pairs, so a run resumes. `tools/score-docx-comparators.sh` scores each tool on all pairs, on completed pairs only and per look; `tools/summarize-docx-comparators.py` makes the tables. The engine's own row is scored the same way once its Word package writes comparisons: `score --cmp DIR --blocks docx/` and the name `onediff` in the script. ### Real DOCX pairs `tools/fetch-real-pairs.py --only docx --root DIR` fetches eight pairs of related Bonterms templates (CC BY 4.0) pinned in `tools/real-pairs-pins.json` with their SHA-256. They carry no labels: the gates are that each tool runs, the output opens, and accept-all and reject-all reproduce the inputs. ### DOCX speed `bench/OneDiff.Bench.Docx` times document in, tracked-changes document out, in process and single threaded, for the in-process tools, on joined gold-set pairs of about 5, 40 and 120 pages (`OneDiff.GoldSets.Docx bench-pairs --data DIR --out DIR --lang en --sizes 1,8,24`). One command runs it with the machine record (CPU, cores, ISA flags, load before and after): ``` dotnet build bench/OneDiff.Bench.Docx -c Release ONEDIFF_DOCX_BENCH=<bench-pairs folder> bench/run-docx-bench.sh OUT ``` The engine's row is one more `[Benchmark]` method in `DocxBench` with the same shape (open both, compare, save). The out-of-process tools' wall time per pair is in their `runs.jsonl` and is information only, since it includes process start. ### Combine and chain The combine set is built by `OneDiff.GoldSets generate-combine --out DIR` (seed 20261104, same shuffle as the synthetic set, so the originals are its eval documents; nothing in it trains a model) and scored by `score-combine --data DIR --out FILE [--jobs N] [--resamples N] [--explain CASE]`. `tools/run-combine-gate.sh set|sweep OUT` runs the whole gate on the worker through the queue. **The set.** `combine-v1`, sha256 `5c55ebb558a267e2264c7f84401b79d7d9208ed29bcb29157192abbb92def406`: for each of the 210 eval originals (30 per language, seven languages), three cases with 2, 3 and 4 reviewers. A reviewer makes one to three edits of the span-bearing types (term swap, value amount/date/duration/percentage, clause rewrite, sentence delete, paragraph delete). A case's plan is `shared` (some reviewers replay one random stream, so they make the same edits), `conflict` (some reviewers replay the stream that picks the target but draw their own new value, so they edit the same place differently), `shared+conflict` (four reviewers: two pairs) or `independent`. Insertions are left out of the scored rows (their place in the original is not a span) and are covered by the property sweep. **Gold.** Derived from the labels, not from the plan, so a replay that diverged is still exact. A gold edit is a unit's old spans with its new text; reviewers who made exactly that edit form its reviewer set (shared when there are two or more). The spans are narrowed to the characters that really change (a character diff of the old span and the new text, each range widened to whole words except in Han and kana). Two gold edits of different reviewers conflict by the library's overlap rule (shared character, a point strictly inside a range, the same point) and are chained into gold conflict components. The reference and label changes that a deleted paragraph causes are not labelled; they are left out of the predicted side too. **Rows** (documents are the resampling unit: all cases of one original together): - *attribution accuracy*: of the gold edits, the share for which every predicted edit that overlaps it and names one of its reviewers names exactly its reviewer set; - *shared-edit merge recall / precision*: the same test over gold edits with two or more reviewers / the share of predicted edits with two or more reviewers that equal the reviewer set of a gold shared edit they overlap; - *conflict recall*: the share of gold conflict components touched by a predicted conflict at two or more of their edits; *conflict precision*: the share of predicted conflicts (not made only of cascade changes) that touch two gold edits of one component; - *invertible rate*: the share of cases whose combination verifies (`Check()` empty, no error diagnostic) at fallback level 0 with every reviewer's comparison at level 0. **Gates.** The seeded property sweeps are the tests `RandomReviewersAlwaysVerify` (1 to 4 reviewers per random document tree, with shared mutations replayed from one stream; `ONEDIFF_COMBINE_CASES`, `ONEDIFF_COMBINE_SEED`) and `RandomChainsAlwaysVerify` (3 to 5 steps; `ONEDIFF_CHAIN_CASES`, `ONEDIFF_CHAIN_SEED`). The cross-machine digest is `digest-combine --data DIR --count 20`: it hashes the combination JSON and report page of the first 20 cases, twice. Timing is `time-combine`: a combine of 4 reviewers on a 100-page synthetic contract. Results: `results/combine-v1-eval.md`. ## Scan gold set scan-v1 is the digital gold set's Latin-script eval pairs scanned under seeded degradations; `tools/OneDiff.GoldSets.Scan` builds it (`build`, `manifest`), runs the exact gates (`gate`: Verify, fallback, JSON round trip, marks re-read on both sides, determinism, a digest for cross-platform comparison) and times the cost (`time`). Rows come from the digital scorer with `--fuzzy-blocks` (recognized text is matched to the gold blocks by similarity and positions carry over by a page-wide character alignment) plus `noise.changes.share`, `noise.changes.per10kchars` and `ocr.flag.P/R/share`. Method in `ocr.md`, results in `results/scan-v1-eval.md`. ## Budgets `bench/run-budgets.sh OUT` times the document budgets with the machine record (`bench/record-run.sh`: CPU, cores, instruction-set flags, load before, during and after). Each part runs in its own process with `DOTNET_PROCESSOR_COUNT=4`, one untimed warm-up and three timed repetitions, and the table reports the median: - the PDF pair of 100 pages and about 300 changes (the test generator `BudgetTests` in `tests/OneDiff.Pdf.Tests`, seed 2026, written once to `OUT/pair` so that the peak memory of a timed part does not include writing the inputs): read both, compare, mark the new document's pages; - the same pair to a rendered redline, inline and side by side, with the peak working set of each process; - a 100-page DOCX contract with about 300 changes to a tracked-changes DOCX (`TimingTests` in `tests/OneDiff.Word.Tests`; information only); - bytes per page: the peak resident set of the part above that of a process that only loads the same inputs (the `baseline` part), divided by the pages. The peak is the operating system's own maximum resident set size for each process: `/usr/bin/time -l` on macOS, `/usr/bin/time -v` on Linux (or `VmHWM` where GNU time is missing). The .NET process counters are not used, because they read 0 on macOS. A run that cannot measure the peak prints `unmeasured` and exits non-zero rather than print a zero or a negative figure. `ONEDIFF_BUDGET_PAGES` and `ONEDIFF_BUDGET_REPS` shrink a run for a quick check; such runs are indicative. Budgets are proposals: PDF pair 2 s; rendered inline 2 s with a peak of at most 1 GB.  # Building, CI and releases ## Building ``` dotnet build OneDiff.slnx -c Release dotnet test --solution OneDiff.slnx -c Release --no-build ``` The SDK version is pinned in `global.json`. Build output goes outside the checkout: set `ONEDIFF_ARTIFACTS` to choose the folder. Without it, output goes to `onediff/build` under the user's local application data folder (`~/.local/share` on Linux and macOS, `%LOCALAPPDATA%` on Windows), or under the temp folder when there is no such folder. ## Sister-library previews The released packages (`RecursiveAI.OneDiff`, `.Documents`, `.Pdf`, `.Ocr` and `.Tool`) depend only on packages published on nuget.org: OneFont 1.4.0, OnePDF 1.18.0, OneImage 1.10.5, OneRender 1.3.3 and oneocr 1.0.0. A fresh clone with nothing configured builds and tests them. The packages that ship later in 1.x (`.Word`, `.Spreadsheets`, `.Presentations`, `.Rendering`, `.Nlp`, `.All`) and some test and tool projects depend on a prerelease of another Recursive AI library that is not on nuget.org yet (OneDocument with OneArchive and OneCompress, OneNLP). The pins are the prerelease `RecursiveAI.*` entries in `Directory.Packages.props`. Building or testing one of those projects with nothing configured stops at restore with a single message naming the project and the packages, for example: ``` OneDiff.Word needs sister-library previews that are not on nuget.org yet: RecursiveAI.OneDocument 0.2.0-preview.2. Set ONEDIFF_PREVIEW_FEED to a folder or feed that holds them, then restore again (see docs/building.md). ``` **To restore them,** set `ONEDIFF_PREVIEW_FEED` to a folder or feed that holds the `.nupkg` files; `Directory.Build.props` adds it as one more restore source, so no other setup is needed: ``` ONEDIFF_PREVIEW_FEED=/path/to/previews dotnet build OneDiff.slnx -c Release ``` The other ways in: a `NuGet.Config` at the repository root (copy `docs/nuget-local.config.example`; the path is not tracked; the message above is not shown when the file exists and it maps the previews to a folder), or `-p:OneDiffPreviewsConfigured=true` when a source is configured elsewhere, such as the user-level NuGet config. Once those libraries are published, the pins move to the released versions and none of this is needed (the CI selector reads the pins too). ## CI `.github/workflows/ci.yml` runs on every push and pull request: - **build**: build and test on linux-x64, linux-arm64, osx-arm64 and win-x64; publish and run the NativeAOT smoke app (every package's main path: text diff, documents, PDF comparison with on-page markup and rendered output, scans, JSON, HTML, combine) and the NativeAOT `onediff` tool on each with warnings as errors. - **pack**: packs every packable project as a workflow artifact (nothing is pushed anywhere). - **wasm**: publishes the test sources of the core, `.Documents`, `.Pdf` and `.Ocr` as browser WebAssembly apps (`tests/OneDiff.Tests.Wasm`, one per suite) and runs each in headless Chrome. A test that needs files, threads, processes or a native oracle carries `[Trait("Host", "Native")]` (or `Assert.SkipUnless`) and is counted as skipped; the run fails on any other failure. Locally: `tests/OneDiff.Tests.Wasm/run.sh <output-folder> [Core Documents Pdf Ocr]`, which needs the `wasm-tools` workload and `uv` or a Python with `playwright`. - **plan**: decides what the jobs above build, from the preview pins. Hosted runners cannot restore a prerelease that is not on nuget.org, so `plan` runs `tools/ci/select-solution.py`: - With no prerelease pins, or with the repository variable `ONEDIFF_PREVIEW_FEED` set to a feed the runners can reach, everything is built (`OneDiff.slnx`) and the preview config above is written into each job. - Otherwise the jobs build `OneDiff.Hosted.slnf`, generated on the fly: every project of the solution that does not depend, directly or through project references, on a prerelease package. The run prints a notice naming the projects it left out, so a green run never silently covers less than it says. The AOT smoke app runs only if it is among the projects built. ## Releases A package **packs** when its project is under `src/` and `IsPackable` is true: `Directory.Build.props` makes everything non-packable, `src/Directory.Build.props` makes projects under `src/` packable, and a project whose package has no content yet sets `IsPackable` to false until it does. A local `dotnet pack OneDiff.slnx` packs all of them. A package is **published to nuget.org** only when its project also sets `PublishToNuGet` to true (`src/Directory.Build.props` sets it to false). That is the one list of what the release publishes: `RecursiveAI.OneDiff`, `.Documents`, `.Pdf`, `.Ocr` and `.Tool` (the `onediff` command), the packages whose dependencies are all on nuget.org once the sister libraries are released. The release also checks that the packed set is exactly these five and that each carries both licences, the third-party notices and its readme. `EnablePackageValidation` is on in every one of those projects. `.github/workflows/release.yml` builds and tests the solution that restores without prerelease packages, packs each published project, and checks that it produced exactly one package per project marked `PublishToNuGet`. A package joins a release by setting the property; it must not depend on a prerelease. Each package has its own `PACKAGE.md` beside its project; it is the readme shown on nuget.org, so its links are absolute. All packages and the tool share one version, `<Version>` in `Directory.Build.props`. The workflow runs when a `vX.Y.Z[-suffix]` tag is pushed and has three jobs: 1. **verify**: the tag is well formed, its commit is on `main`, CI passed on that commit, and `CHANGELOG.md` has a `## [X.Y.Z]` section. 2. **publish**: every packable project builds the tagged version; the solution is built and tested; the packages are packed from those binaries, attested, and pushed to nuget.org with `--skip-duplicate` (the `.nupkg` files first, then the `.snupkg` symbol packages) using Trusted Publishing, so no API key is stored. 3. **release**: a GitHub release with the changelog section as notes and the packages attached. A version with a suffix (`-preview.1`) is marked as a pre-release. Every action is pinned to a commit. The workflow also honours `ONEDIFF_PREVIEW_FEED` for restoring previews that are not published. ## Packing locally ``` dotnet pack OneDiff.slnx -c Release -o <folder> ``` Packs are built with `Deterministic`, and with `ContinuousIntegrationBuild` when `CI=true`. Every entry of a package (the assemblies, documentation files, licence files and the `.nuspec`) has the same contents on repeated builds. The `.nupkg` and `.snupkg` files as a whole do not hash the same, because NuGet stamps each zip entry with the file's modification time and the pack time. Compare two packages by their entries, not by the file hash: ``` for f in a.nupkg b.nupkg; do unzip -p "$f" | sha256sum; done ```  # Changelog All notable changes to this project are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project follows [Semantic Versioning](https://semver.org/). ## [Unreleased] ## [1.0.1] - 2026-11-05 ### Fixed - `RecursiveAI.OneDiff.Documents`: an unsupported Office input or output (DOCX, XLSX, PPTX) now says that support is coming in a 1.x release and suggests exporting the documents to PDF and comparing the PDFs, instead of naming packages that are not on nuget.org yet; the message starts with a capital letter. - `RecursiveAI.OneDiff.Documents`: attribute changes in HTML and the rendered PDF redline show a readable label and unquoted values (`List number: 1.5 → 1.2`) instead of the internal key; `data-key`, `data-old` and `data-new` and the JSON are unchanged. - `RecursiveAI.OneDiff.Documents`: the cover-page line "Hidden into the list below" reads "Not marked (listed below)". - Documentation: the released sister-library versions, the worst speed row (1.13×), and a known issue: scans cannot be read in browser WebAssembly yet. ## [1.0.0] - 2026-11-05 The first stable release of `RecursiveAI.OneDiff`, `RecursiveAI.OneDiff.Documents`, `RecursiveAI.OneDiff.Pdf`, `RecursiveAI.OneDiff.Ocr` and the `onediff` tool (`RecursiveAI.OneDiff.Tool`). Results: [docs/results.md](https://recursiveai.org/onediff/docs/results/). ### Added - `RecursiveAI.OneDiff`: `TextDiff.Pieces` and `TextDiff.Lines` read a diff as typed runs and lines with line numbers; `Diff.SideBySide` lays a line diff out in aligned rows with the changed words marked; `DiffSettings` (algorithm and work cap) is taken by the `Diff` one-liners and `TextDiff.Diagnostic` reports the cap; `Diff.ApplyUnified`, `ApplyPatchText` and `PatchText` take the apply options, patcher and cancellation token; `TextMerge.Conflicts` and `ItemMerge<T>.Items` expose a merge's conflicts and merged items. - `RecursiveAI.OneDiff`: `HunkResult.Line` and `DmpPatchResult.Location` are `int?` (null when not placed), and `DmpPatcher.Match` became `TryMatch`. - `RecursiveAI.OneDiff`: the `*Using` comparer one-liners (`TextUsing`, `LinesUsing`, `WordsUsing`) and `Diff.Semantic` and `Diff.Efficient` take `DiffSettings`, so the work cap reaches every diff one-liner. `TextDiff.Diagnostic.Work` includes the items of the shared head and tail, so a small real diff no longer reports zero. - `RecursiveAI.OneDiff.Documents`: `Redline.CreateAsync`, `Comparison.OfAsync`, `SaveAsync`, `ComparisonJson.ReadAsync` and `WriteAsync` work on streams that refuse synchronous I/O (for example ASP.NET Core request and response bodies), and every `SaveAs` overload takes a `CancellationToken`. - `RecursiveAI.OneDiff.Documents`: `VerificationResult.VerifiedHere` says whether the completeness checks ran in this process on this value; `ComparisonFormatException` carries the position of a JSON syntax error (`LineNumber`, `BytePositionInLine`, `Path`). - `RecursiveAI.OneDiff.Documents`: `CompareDocument.Origin` holds the bytes a document was read from (and the password), so a PDF is marked from the document itself; `PdfRedline.Auto` chooses from the new document's source format and its origin. - `RecursiveAI.OneDiff.Documents`: `Redline.Register(source, priority)` and `DocumentSources.Register(source, priority)`: a source with a higher priority is asked first, so the order of registration no longer decides which source reads a file. `OcrRedline.Register()` registers the scan source above the PDF source. - `RecursiveAI.OneDiff.Documents`: `RedlineStyle.MarkWholeWords` widens marks inside a word to the whole word, as the page markup and the change list already do. - `RecursiveAI.OneDiff.Documents`: plain text reads leading clause markers (such as `1.1` or `(a)`) as list labels, with the rule the PDF source uses, so renumbering in plain text groups as a cascade the way it does for PDF. - `RecursiveAI.OneDiff.Tool`: `onediff --version`, `--exit-code`, `--whole-words` and `--form`; the tool registers the scan source and reads scans. In packages that are not on nuget.org yet, because they depend on previews of other libraries (`RecursiveAI.OneDiff.Word`, `RecursiveAI.OneDiff.Nlp`): - Word packages: the document library OneDocument (formerly OneOffice) is now referenced as `RecursiveAI.OneDocument*` 0.2.0-preview.2. - `RecursiveAI.OneDiff.Word`: `DocxRedline` writes a comparison as a DOCX with real tracked changes, built on the new document (default) or the old one: `w:ins`/`w:del` for text, paragraphs, rows and cells, `w:moveFrom`/`w:moveTo` with range markers, property changes for paragraphs, tables, rows, cells and sections. Accepting every revision gives the new document and rejecting every one the old. The author is a setting, no date is written unless asked, ids are numbered in document order, and the output is byte-identical from run to run. Format-only edits are run property changes, each change is written as one replaced unit, splits and merges use paragraph marks, changed field results and edited text boxes are tracked in place, and the source reads document properties, the list labels of every story and text boxes through the hub (OneDocument 0.2.0-preview.2). Differences a revision cannot carry are returned in `DocxRedlineResult.NotCarried`. `WordRedline.Register()` makes `Redline.Create` and `SaveAs` use it for `.docx`. See `docs/word.md`. - `RecursiveAI.OneDiff.Word`: `WordDocumentSource` reads DOCX, DOCM, DOTX and DOTM through the OneDocument document hub into the compare model: sections with page setup, headers and footers, paragraphs and list items with resolved numbering labels, tables with merges, footnotes and endnotes, comments, fields (code and result), pictures (content hash, and a perceptual hash for PNG), text boxes, content controls, hyperlinks, resolved run and paragraph formatting, document properties and opaque parts. Anchors name the element and the run of every node and text segment, in the offset space of OneDocument's revision authoring. Tracked changes already in an input are accepted by default, or rejected, or refused (`WordRevisionMode`). Register with `WordRedline.Register()`. See `docs/word.md`. - `RecursiveAI.OneDiff.Word`: `DocxRedline` writes a tracked-changes DOCX for a PDF, a scan, plain text or any other flowing source too. A side that is not a Word file is reconstructed as a DOCX from its compare model (paragraphs, list items with their labels, tables, one header and one footer, character formatting), on the other side's package when that one is a Word file, and labelled as reconstructed in its core properties and by a note in its header (`DocxRedline.ReconstructionNote`); the revisions are then written as for two Word files. The comparison is the one that was verified. See `docs/word.md` and `docs/results/pdf-docx.md`. - `RecursiveAI.OneDiff.Word`: a paragraph whose list is numbered by different definitions in the two packages now gets a property change (the same number can mean different labels in two packages), and blocks that followed a block content control that only the other document has are placed after the copies of what stood inside it, not before. - `RecursiveAI.OneDiff.Nlp`: `NlpValueRecognizer`, an `IValueRecognizer` built on OneNLP's recognizers. It reads written-out English values (`forty-two`, `six percent`, `thirty (30) days`, `two hundred thousand dollars`, `1 year and 6 months`) into the same canonical values as the built-in recognizer, and leaves everything else (other languages, party names, unresolved dates) to it. `CompareSettings.Legal.UseOneNlp()` selects it for one comparison and `OneNlpRecognizers.Register()` for the process. See `docs/nlp.md`. ### Changed - **Breaking:** `ComparisonJson.Read` and `ReadAsync` re-run the completeness checks by default (`verify: true`), so `Verification.Passed` is true only when this process proved it, and a tampered file shows failures. Pass `verify: false` for the cheaper read: the value then reports `VerifiedHere == false` and `Passed == false`, and `Comparison.Check()` returns the failures without changing it (a `Comparison` is immutable; read again with `verify: true` for a proven value). - **Breaking:** a PDF with pages but no text is refused (`InvalidDataException`, `PdfSourceErrors.FailureOf` = `NoTextLayer`, with a message naming `RecursiveAI.OneDiff.Ocr`) instead of being compared as an empty document; a PDF whose pages have no content at all is refused as `Malformed` and described as damaged, with or without the scan source registered. Content OnePDF could not recover becomes the comparison diagnostic `source.<side>.pdf.unrecovered`, which strict settings throw on. A scan in which the OCR engine finds no word carries `source.<side>.ocr.nothingRecognized`, with the same strict behaviour. - **Breaking:** `PdfRedline.Auto` and `SaveAs(".pdf")` no longer consult process-wide state. A comparison read back from JSON has no PDF to mark and says so (use `PdfRedline.Rendered`, or `MarkNewDocument` with the PDF's bytes). - `ItemMerge` is an unsealed base class of `ItemMerge<T>`. - `RecursiveAI.OneDiff`: the line type of `TextDiff.Lines` and `TextDiff.Pieces` is named `DiffRun` (it was `DiffPiece`, which clashed with DiffPlex's type in files that import both during a migration). - `RecursiveAI.OneDiff.Documents`: plain-text comparisons now carry list items, so the JSON of existing text comparisons changes where lines start with clause markers. - Exception messages start with a capital letter. ### Removed - `RecursiveAI.OneDiff.Documents`: `Redline.Compare`, the alias of `Comparison.Of`. Use `Comparison.Of`. ## [0.2.0-preview.1] The first release on nuget.org: `RecursiveAI.OneDiff`, `RecursiveAI.OneDiff.Documents`, `RecursiveAI.OneDiff.Pdf` and `RecursiveAI.OneDiff.Tool`. A first preview, 0.1.0-preview.1, was never published; its contents are included below under that heading. The public API may change before 1.0. ### Added - `RecursiveAI.OneDiff.Documents`: change semantics. `Change` gains `Type` (`ChangeType`: amount, date, duration, percentage, number, party, defined term, reference, numbering, formatting, typography), `OldValue` and `NewValue` (`ResolvedValue`: canonical, language-neutral forms such as `USD 25000`, `2027-03-01`, `30 d`) and `IsSubstantive`. With `CompareSettings.CollapseCascades`, `Stages.Group` lifts a renumbering and the references it caused, and a renamed defined term and its occurrences, into `GroupKind.Cascade` and `GroupKind.Rename` display groups; every underlying change stays in the list. With `TypedChanges`, `Stages.Classify` types and resolves changes. New public seam `IValueRecognizer` (`Stages.Classify(changes, recognizer)`), with `BuiltInValueRecognizer` as the default for the seven measured languages. The JSON gains optional `type`, `oldValue`, `newValue` and `substantive` members on changes (additive in schema version 1); the HTML pages list each cascade and rename group with its occurrences. See `docs/semantics.md`. - `RecursiveAI.OneDiff.Documents`: combine, version chains and the change report (`OneDiff.Documents.Combine`). `Combination.Of(original, copies)` compares every reviewer's copy with the original and combines the results: each change is attributed to the reviewers who made it, identical changes are merged (and each edit is attributed on its own), and changes that cannot all be applied are listed as conflicts with every version shown, never a winner. The changes attributed to a reviewer are exactly that reviewer's verified comparison, checked on every run (`Combination.Check`, with a fallback that lists every change on its own). `ComparisonChain.Of(versions)` verifies every step and the net comparison and says which steps touched each net change (`StepsOf`). `ChangeReport.From(comparison | combination | chain)` gives counts by kind and type, the substantive list with resolved values, conflicts and per-step counts, and renders one self-contained HTML page. `Redline.Combine(original, copies, output)` and `Redline.Chain(versions, output)` are the one-liners; the JSON forms are `onediff.combination` and `onediff.chain` (version 1, schemas in `docs/schema`), which embed unchanged `onediff.comparison` documents. See `docs/combine.md`. - `RecursiveAI.OneDiff.Tool`: `onediff compare` reads PDFs and writes `.pdf` output (the new PDF with the changes marked on its pages, or a rendered redline for other inputs), and takes `-o <output>`; the tool references `RecursiveAI.OneDiff.Pdf`. Rendered redlines draw Latin text only (no script font packages). - `RecursiveAI.OneDiff.Tool`: `onediff combine <original> <copy>... -o <out>` and `onediff chain <v1> <v2>... -o <out>`. - `RecursiveAI.OneDiff.Documents`: `CompareSettings.ValueRecognizer` and `ValueRecognizers.Register` choose the value recognizer of a comparison (settings, else the registered one, else the built-in one); the fallback stages read it from the settings too. Not written to JSON. - `RecursiveAI.OneDiff.Pdf`: the rendered redline. `PdfRedline.Rendered` and `PdfRedline.RenderedSideBySide` (`RenderedPdfRedline`, with `Paper` and `Render`) write a tagged PDF laid out from any comparison, plain text included: the cover page, key, marks, ledger and footer of the HTML redline set on A4 or Letter with a running page number, in any `RedlineStyle`, byte-identical across runs. It depends on `RecursiveAI.OneRender.Pdf`. - `PdfRedline.Auto`, registered for `.pdf` by `PdfRedline.Register()`: original-page markup when the new document is a PDF, a rendered redline otherwise. Both forms can still be named. - Release packaging: the release workflow publishes the projects that set `PublishToNuGet`; a local `dotnet pack` still packs every packable project. Each package has its own readme. The OnePDF and OneFont pins are two properties in `Directory.Packages.props`. - Sample PDFs for every preset and layout, and one original-page markup sample, under `docs/samples`, with a README of the one-line recipes. ### Changed - `PdfRedline.Register()` registers `PdfRedline.Auto` for `.pdf` instead of `PdfRedline.OnPages` alone; a PDF as the new input is marked on its pages as before. ### Not in this release - `RecursiveAI.OneDiff.Word` (DOCX source and tracked-changes writer) and `RecursiveAI.OneDiff.Nlp` (OneNLP value recognizer) are in the repository but not published, because they depend on previews of OneDocument and OneNLP that are not on nuget.org. Change semantics, combine, chains, the change report and the recognizer seam live in `RecursiveAI.OneDiff.Documents` and are in this release. - Scans, spreadsheets, slides, the model and the OneDocument rendering package are not built yet. - `RecursiveAI.OneDiff.Ocr`: `OcrDocumentSource` reads scans into the flowing compare model through the local OCR engine of `RecursiveAI.OneOcr`: images (PNG, JPEG, TIFF, recognized by content) and PDFs whose pages carry no text of their own (`OcrDocumentSource.IsScannedPdf`). Paragraphs, list items (clause numbers included), and running headers and footers (found by their repetition across pages) become nodes whose words are anchored by `AnchorKind.OcrWords`: the page quadrilateral in upright page space and the engine's confidence. `OcrSourceOptions` carries the engine, resolution, limits and layout switches. `OcrRedline.Register()` registers the PDF and scan sources and the on-page markup form; a scan is marked on its own pages (an image on a PDF page made from it) by `PdfRedline`. See docs/ocr.md. - `RecursiveAI.OneDiff.Documents`: possible OCR errors. `Change.OcrConfidence` is the lowest confidence of the recognized words a change's edits touch, and `Change.PossibleOcrError` is true when it is below `CompareSettings.OcrConfidenceThreshold` (default 0.95, chosen on a development scan set). The comparison JSON carries `ocrConfidence` and `possibleOcrError` on changes and `ocrConfidenceThreshold` in the settings when it differs from the default; all three are optional additions to schema version 1. - Gold sets: scan-v1, the Latin-script pairs of the digital gold set scanned under seeded degradations (`tools/OneDiff.GoldSets.Scan`), with exact gates, noise and flag rows and an indicative cost row. See docs/results/scan-v1-eval.md. - `RecursiveAI.OneDiff.Pdf`: the markup writer places marks on `AnchorKind.OcrWords` anchors as it does on glyph quadrilaterals. ## [0.1.0-preview.1] The first preview. Public API may change before 1.0. ### Added #### `RecursiveAI.OneDiff` (text core, no dependency beyond the .NET base class library) - Edit scripts as struct ranges (`Edit`, `EditScript`, `EditScriptBuilder`), `TokenInterner<T>`, and the `Diff` entry points for lines, words, characters and graphemes, with comparer-based variants (ignore case, ignore whitespace). - Myers (linear space, with a work cap and a `DiffDiagnostic` when it is reached), patience and histogram diff over any sequence with an equality comparer. - Line, word, character and grapheme tokenizers; word tokens are script-aware, so Han, Hiragana and similar scripts split per character. - Semantic and efficiency cleanup, with lossless shifting of changes to word, sentence and line boundaries. - Unified diff: make, parse and apply with offset and fuzz, reporting hunks that do not apply. - diff-match-patch patch text: make, parse and apply with fuzzy matching, with UTF-16 or code-point offsets. - Three-way (diff3) merge with `merge`, `diff3` and `zdiff3` conflict styles, resolution options and a structured conflict result. - Terminal and Markdown renderers for line and word diffs. #### `RecursiveAI.OneDiff.Documents` (document comparison engine) - The compare model: block trees with source anchors, formatting runs and normalization folds; `IDocumentSource` with a plain-text source. - The five stages (`Align`, `DiffPairs`, `Group`, `Classify`, `Verify`) over immutable values, an editable `Alignment`, and deterministic block alignment with moves, splits, merges and table-aware matching. - `Comparison` with a change ledger, diagnostics and `Verify`: every comparison is checked for completeness (old plus changes gives new, new minus changes gives old) and falls back stepwise to a coarser complete grouping, or throws in strict mode. - `CompareSettings` and `RedlineStyle` presets, and a versioned JSON form (schema v1) with a published schema. - Self-contained HTML redlines, inline and side by side, with cover page, change bars and footer; the `Redline` statics (`Create`, `Compare`), `Comparison.SaveAs`, `Save` and `ToBytes`, and a registration seam (`DocumentSources.Register`, `Redline.Register`, `RedlineForm`) for adapter packages. #### `RecursiveAI.OneDiff.Pdf` (PDF source and original-page markup) - `RecursiveAI.OneDiff.Pdf`: `PdfDocumentSource` reads digital PDFs into the compare model (paragraphs joined across lines, columns and pages, list items, tables, running headers and footers, formatting, glyph-quad anchors; line breaks, hyphenation and page numbers go to the ledger) and `PdfRedline` marks a comparison on the original pages with real highlight, caret, underline, squiggly and strike-out annotations carrying the old text in their pop-ups. See `docs/pdf.md`. #### Repository - CI on linux-x64, linux-arm64, osx-arm64 and win-x64, a NativeAOT smoke app, and the core tests run as a browser WebAssembly app in headless Chrome. - A release workflow that publishes through nuget.org Trusted Publishing; see `docs/building.md`. ### Changed - Building without `ONEDIFF_ARTIFACTS` set no longer fails on the default output folder.