---
title: "Redaction: completeness review"
url: https://recursiveai.org/onedocument/docs/redaction-review/
description: "The project's own review (not an external audit) of Office.Redact / OfficeFile.Redact. The question: after a match or region is redacted, can the removed…"
version: "1.0.0"
source: https://github.com/recursive-ai-oss/onedocument-csharp/blob/a85720e075c783fc596641d8586ee126cae088c5/docs/redaction-review.md
---

# Redaction: completeness review

The project's own review (not an external audit) of `Office.Redact` / `OfficeFile.Redact`. The question: after a match or region is redacted, can the removed text be recovered from the output file by any means? The method was to list every place a string can live from the format specifications, compare the list with `docs/redaction.md`, attack the feature with hostile inputs (generated by `tests/OneDocument.Redaction.Tests/Attacks.cs`), and test the verifier on the survivors it should report.

The attacks are read back by an independent reader (`Attacker.cs`) that knows nothing about the redaction code: zip entries, gzip and zlib streams anywhere in a binary, compound-file streams through their sector chains, macro source after decompression, XML text with character references, `_xHHHH_` escapes, percent-escapes to any depth, text joined across elements, base64 tokens, case, accents and invisible characters.

## Verdict

**Pass, with the matching contract stated and the open items below listed in the feature's own documentation.**

After the fixes in this review there is no tested input from which a matched string can be read back and which the report calls verified, except the four where the plan's literal does not match the text at all (invisible characters, another Unicode normalisation form, a no-break space, a typographic apostrophe: matching is exact, see Open items). Every other survivor is either removed or makes `Verified` false and appears in `Survivors` or `Gaps`. Copies of a region's text are reported as `redaction.region-copy` gaps and do not change `Verified`, which speaks for matches.

Before the review the feature failed this standard in ten ways. Each is a regression test now:

| # | Recoverable and reported as verified | Kind | Fix |
|---|---|---|---|
| 1 | The secret in a text node or attribute written as base64 (custom XML blobs, embedded files, signature certificates) | redaction gap and verifier false negative | whole-value base64 is decoded, overwritten in the bytes and written back; the verifier decodes it |
| 2 | SpreadsheetML `_x0051_UOKKA` escapes in shared strings, inline strings, string results, names | both | decoded for matching, re-escaped on write (`EncodedText`) |
| 3 | A string split across two sectors that are not neighbours in an embedded compound file (every Office-saved object is fragmented like this) | both | streams are mapped through the sector chains and searched and overwritten stream by stream (`CompoundFileMap`) |
| 4 | Zip comments, archive comment, Info-ZIP unicode name fields carried over from the input | both: untouched entries keep their header metadata, the verifier read entry content only | blanked in the written file (`ZipMetadata`); the verifier searches every header |
| 5 | Compressed text chunks of a PNG (`zTXt`, deflated `iTXt`) | both | a chunk with a match is dropped whole (`PngChunks`); the verifier inflates deflated data inside binary parts |
| 6 | Macro source (compressed), packages stored inside an object stream, gzip parts | verifier false negative (redaction cannot rewrite these) | the verifier decompresses them, so `Verified` is false and `redaction.opaque-part` names the part |
| 7 | An unreferenced `docProps/thumbnail.*` | redaction gap | dropped with the others |
| 8 | A link target encoded twice (`%2551`) | both | decoded as far as it goes, re-encoded the same number of times. `Escape` also no longer leaves `%` alone |
| 9 | A secret that is also a substring of one of the identifier marks (`RED` in `REDACTEDA`), surviving elsewhere, was ignored by the verifier | verifier false negative | a match counts as the redaction's own only when it lies inside a mark written at that place |
| 10 | UTF-16 big-endian text in a binary part (the name table of an embedded font, the strings of an included PDF): neither searched nor verified | both | big-endian needles and runs |

Two further defects in region redaction:

| # | Defect | Fix |
|---|---|---|
| 11 | `SlideRegion` left the slide whole in the file when another slide linked to it (the link kept it reachable) | the links to a removed slide (hyperlink element and relationship) go with it; any other reference is reported |
| 12 | A region removes by position, so copies of its text (table of contents entries, cross-references, cached formula results) stayed, with no report | `redaction.region-copy` gap per part still holding the text; nothing is removed (design question below) |

One regression was found by running LibreOffice over the outputs: the first version of the zip header cleaning zero-filled extra fields, which left a truncated field that LibreOffice refused to open. The cleaner now only rewrites the Info-ZIP name and comment fields and leaves others alone.

## Where a string can live: the independent list

Sources: ECMA-376 Parts 1–4, [MS-DOCX], [MS-XLSX], [MS-PPTX], [MS-ODRAWXML], [MS-OFFMACRO2], [MS-OVBA], [MS-OFFCRYPTO]. Status per location:

- **T** covered and tested (a row in `Attacks.cs` or the torture packages);
- **U** covered by the generic text/attribute pass over every XML part but without its own test;
- **D** gap, documented in `docs/redaction.md`;
- **N** gap that was not documented; fixed or documented by this review.

Every XML part of the package and of each package embedded in it goes through the same two passes (paragraph-like units across runs; every other text node, attribute value, comment and processing instruction), so a location in an XML part is **U** unless it needs a rule of its own.

### Package level

| Location | Status |
|---|---|
| Part names; names in `[Content_Types].xml`; internal relationship targets | D (`redaction.part-name`; the verifier reports a survivor) |
| External relationship targets (hyperlinks, linked files, mail merge sources, external links, attached templates), percent-encoded to any depth | T (N for double encoding, fixed) |
| Relationship identifiers and types | D (added: not rewritten, reported) |
| Zip archive comment, entry comments, Info-ZIP unicode name/comment fields, local headers | N, fixed (T) |
| Entries that are not parts; unreferenced parts | T (verified, redacted as parts) |
| Preview images `docProps/thumbnail.*`, referenced or not | T (the unreferenced case was N, fixed) |
| `docProps/core.xml`, `app.xml`, `custom.xml` (names, values, company, manager, titles of parts) | T |
| Flat OPC input (`pkg:binaryData` parts) | T |
| Encrypted input (OOXML Agile, binary RC4) | T for the content; the output is not encrypted (D, added) |
| Legacy input (DOC, XLS, PPT, RTF, 2003 XML) staged to OOXML | T (information from fixtures and RTF: split groups, `\'hh`, `\uN`, `\info`, hidden text, field instructions) |

### WordprocessingML

| Location | Status |
|---|---|
| Runs (`w:t`), across runs, proofing marks, smart tags, bookmarks, hyperlinks, tracked insertions and deletions, moved text | T |
| Field codes and cached results, `w:fldSimple` | T |
| XML comments, CDATA pieces and character references inside text | T |
| `mc:AlternateContent` Choice and Fallback, including a secret split across the two branches | T |
| Text boxes, shapes, content controls (alias, tag, placeholder, bound value) | T |
| Comments, replies, `people.xml`, `commentsExtensible` extension lists; `commentsExtended`, `commentsIds` (identifiers, no text) | T / U |
| Footnotes, endnotes, headers, footers, glossary document | T |
| `settings.xml`: document variables, attached template, mail merge connection string, query, data source, ODSO | T |
| `settings.xml`: `documentProtection` and `writeProtection` hashes and salts (base64 of random bytes) | U (left untouched by the base64 rule unless a literal is in the bytes; tested with a hash under a digit pattern) |
| Style and numbering names, `lvlText`, fonts table, web settings | U |
| Custom XML items, item properties (`schemaRef uri`), base64 inside items | T (the base64 case was N) |
| Revision attributes (`w:author`, `w:date`), property-change records | U |
| `w:rsid*` values | not text; they fingerprint editing sessions and belong to sanitising |
| Drawings: `docPr`/`cNvPr` name, description, title; picture and shape text; WordArt text; VML `textpath`, `alt` | T / U |
| Charts (caches, titles, user shapes), chartex data (`cx:`) | T / U |
| SmartArt data, cached drawing, layout definition titles and descriptions | T / U |
| Embedded workbooks and documents (packages) | T |
| Embedded objects (`.bin` compound files, package streams, macro projects) | T for plain streams (N for fragmented, fixed); D for compressed content |
| Included files (`altChunk`): HTML with character references | T; RTF and mail archives: D (verified, not rewritten) |
| Embedded pictures: metadata chunks and EXIF | T (PNG text N, fixed); pixels D |
| `customUI` ribbon parts, `w:customXmlPr` | U |
| Ink parts (InkML annotations) | U |
| Signature parts (`_xmlsignatures`): certificates and signature properties in base64 | T through the base64 rule; any edit invalidates the signature (D, added) |
| VBA project `vbaProject.bin`, `vbaData.xml` | D (compressed source is verified, not rewritten) |

### SpreadsheetML

| Location | Status |
|---|---|
| Shared strings, rich runs, phonetic runs, inline strings, string results, with `_xHHHH_` escapes | T (escapes N, fixed) |
| Numbers and formula results that match | T |
| Formula text and string literals; defined names and their `comment`; sheet names used in other sheets' formulas | T |
| Table names, column names, calculated column formulas | T |
| Data validation titles, prompts, errors and list formulas; conditional formats including extension lists (`x14`) | T |
| Headers and footers, print titles | T / U |
| Legacy notes, their VML drawing, threaded comments, `persons` | T |
| Hyperlinks (location, display, tooltip, targets) | T / U |
| Pivot caches (definition, records, shared items), pivot tables, slicer and timeline caches | T (caches) / U |
| External link caches, sheet names, paths | T |
| Connections, query tables (connection strings, commands) | T |
| Rich values (`richData`) and metadata | T |
| Custom number formats with literal text (`formatCode`) | U |
| Form controls and ActiveX (`ctrlProps`, `activeX*.bin`) | U / D (binary property bags are searched as bytes and streams) |
| Printer settings (`.bin`, UTF-16 device names) | T |
| Cell values of a range removed by a region: charts, formulas elsewhere | D (derived values) and now `redaction.region-copy` |

### PresentationML

| Location | Status |
|---|---|
| Slide text, notes, notes master, handout master, slide masters and layouts, layout and slide names, custom shows, section names, tags | T |
| Comment authors and comments, modern comments and authors | T |
| Alt text and descriptions (`descr`, `adec:decorative` extension), hyperlink tooltips and targets, action links | T / U |
| Media and embedded font names (part names) | D |
| Charts, tables, SmartArt, OLE objects | T / U |
| Animations and transitions text (`p:txEl`, build paragraphs) | U |
| `docProps/app.xml` slide titles | T (dropped by a slide region) |
| Region: a slide that other slides link to | N, fixed (T) |

## Attacks and results

`Survived` means the independent reader recovered the secret from the output and the report did not say so. `Reported` means it is recoverable but `Verified` is false and the report names it. `Removed` means nothing is recoverable.

| Attack | Result |
|---|---|
| Secret split across runs, proofing marks, smart tags, bookmarks, hyperlinks, revisions (ins, del, move) | Removed |
| Secret split between a field's code and its result runs | Removed |
| XML comment, CDATA pieces, character references inside text and attribute values | Removed |
| `mc:AlternateContent` Choice and Fallback, secret split across both | Removed |
| Unknown-namespace element, attribute, comment, processing instruction; `extLst` content | Removed |
| Attribute with a prefix other than `r:` bound to the relationships URI elsewhere | Removed |
| Namespace URI, element name, attribute name, relationship id holding the secret | Reported |
| Secret as a case variant (`ignoreCase`) | Removed |
| Secret as `QUO` + U+200B + `KKA` + U+00AD + `4419`; decomposed accents; no-break space for a space; typographic apostrophe | Survived by design: the literal does not match the text (see Open items) |
| Custom XML with base64 (UTF-8 and UTF-16 payloads) | Removed (was Survived) |
| Custom XML item properties, people parts, extension lists | Removed |
| Attached template, mail merge connection, query, data source and ODSO relationships | Removed |
| Header with a field code; sdt placeholder; core properties | Removed |
| Orphan thumbnail | Removed (was Survived) |
| VBA project whose source holds the secret behind a copy token | Reported (was Survived and verified); `redaction.opaque-part` |
| Embedded compound file with the secret across two non-adjacent sectors, ASCII and UTF-16 | Removed (was Survived and verified) |
| Compound file holding a zip with the secret deflated | Reported (was Survived and verified) |
| PNG with the secret in a deflated text chunk | Removed (was Survived and verified) |
| gzip part | Reported (was Survived and verified) |
| UTF-16 XML part, plain and with character references | Removed |
| Embedded font whose name table (UTF-16 big-endian) holds the secret | Removed (was Survived and verified) |
| XML part declared in a legacy encoding | Reported |
| Link target percent-encoded, doubly encoded | Removed (double was Survived) |
| Link target in a `HYPERLINK` field code | Removed |
| Part name, content-type default and override, relationship target naming the secret | Reported |
| Included HTML with character references | Removed |
| Included RTF with `\'hh` escapes; included quoted-printable mail archive | Reported (before, the verifier searched bytes only and would have called both clean) |
| Zip entry comments and archive comment; Info-ZIP unicode path field | Removed (was Survived and verified) |
| Flat OPC input with a binary part | Removed |
| SpreadsheetML `_x0051_` escapes in shared strings, inline strings and string results | Removed (was Survived and verified) |
| Rich text runs, phonetic text, numbers, formulas, sheet names in other formulas | Removed |
| Validation prompts, headers, legacy comments with VML, threaded comments, persons, connections, rich values, slicer caches, external links, pivot caches, table columns, printer settings (UTF-16), `x14` extension formulas | Removed |
| PowerPoint tags, sections, custom shows, comment authors, handout master, layout names, modern comments | Removed |
| Encrypted inputs (Agile docx and xlsx, RC4 doc) | Removed; output is unencrypted |
| Legacy inputs staged to OOXML (DOC, DOC with notes, XLS, PPT, RTF) | Removed |
| Verifier ignoring a hit because it is a substring of any mark | Fixed (was a false negative) |
| Paragraph region: table of contents entry and cross-reference result copies | Reported (`redaction.region-copy`; was silent) |
| Cell range region: cached results of formulas elsewhere | Reported |
| Slide region with a jump link from another slide | Removed (slide stayed whole) |

## The verifier

It must not say verified when something recoverable survives. Its false negatives, all now closed unless listed under Open items:

1. Zip headers (comments, extra fields): not read. Now every header region is searched.
2. Compound-file streams across non-adjacent sectors: not read. Now mapped through the chains.
3. Compressed data inside binary parts and streams (zlib at any offset, gzip, macro source, packages in streams): not read. Now decoded, bounded to 2,000 candidate starts per entry and to the part size limit; inside pictures only literal matches count.
4. `_xHHHH_` escapes and base64 values: not decoded. Now decoded.
5. Percent-encoding beyond one level. Now four.
6. Text parts that are not XML (RTF, HTML, quoted-printable): searched as bytes only. Now decoded.
7. A hit ignored because it is a substring of some mark. Now only inside a mark at that place.

What it still cannot see, so `Verified` can be true with the secret recoverable: pixels, text in formats nobody here decodes (PDF streams inside objects are inflated only at zlib starts it finds, and their fonts may encode text through glyph ids), encrypted payloads, and the matching variants under Open items.

## Open items

1. **Matching is exact on characters.** The four skipped tests in `AttackTests` are this. Options: (a) keep, document, tell callers to list variants or use a pattern (current); (b) add an additive `RedactionOptions.Relaxed` that folds no-break and thin spaces to a space, typographic quotes and dashes to ASCII, and ignores soft hyphens and zero-width characters, with an index map so spans map back to the original text; (c) also normalise Unicode forms, which needs normalisation tables of our own because the library runs with invariant globalization. Recommendation: (b) for 1.0 or 1.1, since a no-break space inside a name and an autocorrected apostrophe are common in Word documents; (c) later.
2. **Copies of a region's text** are reported, not removed. Option: remove text copies of at least N characters automatically (as a match), with an option to turn it off. Risk: removes innocent repeats. Recommendation: keep reporting, add `RedactionPlan.RemoveRegionCopies` (additive) when a caller asks.
3. **Part names and relationship identifiers** are not rewritten. Option: rename parts to `/path/REDACTEDA.ext` and rewrite every relationship target, content-type override and Flat OPC name that points to them. Recommendation: do it when a real document shows a secret in a part name (media and attachment file names are the realistic case).
4. **Compressed macro source, packages inside object streams, included RTF and mail archives** are verified, not rewritten. A rewrite of macro source needs a recompressor that fits the old stream length. Recommendation: document that redaction of a macro-enabled file needs the project removed (`docs/vba.md`), and consider making `Redact` accept a "remove macros" option.
5. **Signed documents**: any redaction invalidates the signatures, and the signature parts carry the signer's certificate (a name). The base64 rule now overwrites a match inside a certificate, which only makes the corrupt signature obvious. Recommendation: drop signature parts and report `redaction.signatures-removed`, since a redacted file cannot keep a valid signature.
6. **Office checks** (the output of each new rewrite opens without a repair prompt): the base64 rewrite of custom XML, the `_x` re-escape of a shared string, the dropped PNG chunk, the blanked zip comments and the removed slide links. Committed unanswered in `testdata/redaction/ORACLE-REQUEST.md`.

## Information-only corpus run

Not a gate. Documents from public corpora (never committed), redacted with a pattern and re-verified:

- **Census of 1,850 Office Open XML files** (documents, workbooks and presentations from web crawls and bug trackers). Parts beyond text and pictures: embedded fonts (202), macro projects (76) and one signature, ActiveX controls (44), EMF/WMF pictures (165), video and audio, SVG, one included HTML page, one included Word file, one PDF. Extension namespaces in use go up to Word 2024 (`sdtformatlock`, `word16du`, `sdtdatahash`) and include the three generations of chart extensions (`chartex`), ink, 3D models and the Excel revision streams. The extension namespaces are all covered by the generic text and attribute passes; the embedded fonts led to the big-endian fix (finding 10).
- **Documents, `\bthe\b` ignoring case**: 297 readable files, all redacted and verified (6,499 and 8,910 replacements); 3 truncated downloads gave `format.unrecognised`. 25 files have a `redaction.opaque-part` gap (embedded objects, fonts, pictures).
- **Presentations, same pattern**: 100 of 100 verified.
- **Workbooks, pattern `\b[A-Z][a-z]{5,}\b`**: 198 redacted, 140 verified. The 58 unverified are workbooks with macro projects (the project's module and class names are readable once the source is decompressed: reported, as designed), printer settings that hold deflated text, and the `TargetMode="External"` attribute of relationship files, which a pattern like this matches as structure (see Known gaps in `docs/redaction.md`: patterns should be as specific as the secret).
- Two more truncated workbooks gave `format.unrecognised`. Time per file: 54 to 180 ms on average, 7.7 s at most.