# onedocument docs, all pages (OneDocument 1.0.0) Every page of https://recursiveai.org/onedocument/docs/ as markdown, getting started first. Index: https://recursiveai.org/onedocument/llms.txt # Getting started with OneDocument Install OneDocument 1.0.0 for .NET 10, read a Word document, convert it to PDF, read and recalculate an Excel workbook, and use the `onedocument` command-line tool. Every sample on this page was run on the 1.0.0 packages from NuGet and shows its real output. The program makes its own input files, so you need nothing on disk. OneDocument was called OneOffice before its first release. It is not affiliated with Microsoft; Word, Excel and PowerPoint name the file formats. ## Install The packages target `net10.0`, so you need the .NET 10 SDK. ```shell dotnet new console -o OneDocumentStart cd OneDocumentStart dotnet add package RecursiveAI.OneDocument.Documents --version 1.0.0 dotnet add package RecursiveAI.OneDocument.Spreadsheets --version 1.0.0 dotnet add package RecursiveAI.OneDocument.Rendering --version 1.0.0 ``` Eight packages are on NuGet at 1.0.0, all on one version. Reference only the kinds you use; each one registers its formats at build time, with no setup call. - **`RecursiveAI.OneDocument`**: the core. Containers, the markup tree, the format registry, formulas and number formats, templates, text extraction, RTF and VBA. The others bring it in. - **`RecursiveAI.OneDocument.Documents`**: Word-processing documents: DOCX, DOC, RTF, ODT, 2003 XML, HTML and Markdown, tracked changes, mail merge, operations. - **`RecursiveAI.OneDocument.Spreadsheets`**: workbooks: XLSX, XLSB, XLS, ODS, CSV and TSV, 2003 XML, calculation, pivots, streaming, typed rows. - **`RecursiveAI.OneDocument.Presentations`**: PPTX, PPT and ODP. - **`RecursiveAI.OneDocument.Schema`**: typed views over the OOXML schemas. - **`RecursiveAI.OneDocument.Rendering`**: layout, charts, SmartArt and equations; PDF and page images (PNG, JPEG, TIFF, WebP, SVG). - **`RecursiveAI.OneDocument.All`**: every package above. - **`RecursiveAI.OneDocument.Tool`**: the `onedocument` command-line tool. None of them uses native code. A ninth, `RecursiveAI.OneDocument.Cryptography` (signatures and encryption on write), isn’t on NuGet: in the changelog’s words, “it follows in a 1.x release.” ## Make two files to work with The samples below are one `Program.cs`, run top to bottom. This first part writes a short report and a small budget, so the rest has real files to open. A new document starts with one empty paragraph, as Word’s blank document does. ```csharp using RecursiveAI.OneDocument; using RecursiveAI.OneDocument.Documents; using RecursiveAI.OneDocument.Spreadsheets; using RecursiveAI.OneDocument.Spreadsheets.Binding; using (Document made = Document.Create()) { made.Styles.AddBuiltIn("Title"); made.Styles.AddBuiltIn("Heading 1"); made.Body.AddParagraph("Site visit report").Style = "Title"; made.Body.AddParagraph("Findings").Style = "Heading 1"; made.Body.AddParagraph("The roof drains are clear. The north stairwell light is out."); made.Body.AddParagraph("Next steps").Style = "Heading 1"; made.Body.AddParagraph("Replace the stairwell light by Friday."); made.SaveAs("report.docx"); } using (Document made = Document.Create()) { Paragraph body = made.Body.AddParagraph("The supplier delivers the goods."); made.Revisions.CreateAuthor("Dana Reviewer", null).InsertText(body, 0, "Within 30 days, "); made.SaveAs("draft.docx"); // a tracked change, for the command-line section } using (Workbook made = Workbook.Create()) { Worksheet sheet = made.Sheets[0]; sheet.Rename("Budget"); sheet.Range("A1:B1").Values = new object?[,] { { "Item", "Cost" } }; sheet.Range("A2:B4").Values = new object?[,] { { "Ladder", 189.5 }, { "Bulbs", 42 }, { "Labor", 360 } }; sheet["D1"].SetString("Total"); sheet["D2"].Formula = "SUM(B2:B4)"; sheet.Range("B2:B4").NumberFormat = "$#,##0.00"; sheet.Range("D2:D2").NumberFormat = "$#,##0.00"; made.Recalculate(); // stores the formula’s result in the file, as Excel does made.SaveAs("budget.xlsx"); } ``` ## Read a .docx `Document.Open` finds the format from the content, never the extension. Paragraphs carry their text and style name; `Office.ExtractText` gives the text of any supported file in one call. ```csharp using (Document doc = Document.Open("report.docx")) { foreach (Paragraph p in doc.Paragraphs) Console.WriteLine($"[{p.Style}] {p.Text}"); } string text = Office.ExtractText("report.docx"); Console.WriteLine($"{text.Length} characters: {text.Trim().Split('\n')[0]} ..."); ``` ```text [Normal] [Title] Site visit report [Heading 1] Findings [Normal] The roof drains are clear. The north stairwell light is out. [Heading 1] Next steps [Normal] Replace the stairwell light by Friday. 139 characters: Site visit report ... ``` To change a document in place, edit and call `doc.Save()`: parts you didn’t touch are written back byte-identical. On this report, setting the text of the title paragraph and saving changed `word/document.xml` and nothing else. See [authoring](https://recursiveai.org/onedocument/docs/authoring/) and [tracked changes](https://recursiveai.org/onedocument/docs/revisions/). ## Convert a .docx to PDF One line, with the Rendering package referenced. The PDF is tagged. ```csharp Office.Convert("report.docx", "report.pdf"); byte[] pdf = File.ReadAllBytes("report.pdf"); Console.WriteLine($"report.pdf: {pdf.Length:N0} bytes, starts {System.Text.Encoding.ASCII.GetString(pdf, 0, 8)}"); ``` ```text report.pdf: 14,544 bytes, starts %PDF-1.7 ``` The same call converts a workbook or a presentation, and picks the output by extension: `.png`, `.jpg`, `.tif`, `.webp` and `.svg` write page images, and `.odt`, `.rtf`, `.html` or `.md` convert the document. By default the layout uses the open fonts that come with OneRender (Liberation, Carlito and Caladea), with a metric-compatible stand-in where the document names a font it doesn’t have: Times New Roman becomes Liberation Serif, Calibri becomes Carlito. Your own fonts can be added. See [layout and PDF](https://recursiveai.org/onedocument/docs/layout/) and [page images and SVG](https://recursiveai.org/onedocument/docs/rendering-export/). ## Read an .xlsx Cells give their value, their formula and the text Excel would display. `Recalculate` runs the calculation engine after an edit. ```csharp using (Workbook wb = Workbook.Open("budget.xlsx")) { Worksheet sheet = wb.Sheets["Budget"]; for (int row = 2; row <= 4; row++) Console.WriteLine($"{sheet[$"A{row}"].Text,-7} {sheet[$"B{row}"].FormattedText,9}"); Cell total = sheet["D2"]; Console.WriteLine($"Total {total.FormattedText,9} ={total.Formula}"); sheet["B3"].Value = 58; wb.Recalculate(); Console.WriteLine($"Total {total.FormattedText,9} after B3 = 58"); } ``` ```text Ladder $189.50 Bulbs $42.00 Labor $360.00 Total $591.50 =SUM(B2:B4) Total $607.50 after B3 = 58 ``` For rows of data, `Spreadsheet.Read` binds each row to a record by its header names. A source generator writes the binding code, so mark the type `partial`: ```csharp foreach (Expense e in Spreadsheet.Read("budget.xlsx")) Console.WriteLine(e); [SpreadsheetRow(Sheet = "Budget")] internal sealed partial record Expense(string Item, decimal Cost); ``` ```text Expense { Item = Ladder, Cost = 189.5 } Expense { Item = Bulbs, Cost = 42 } Expense { Item = Labor, Cost = 360 } ``` `Spreadsheet.Write(path, rows)` is the other direction. Formula text is en-US only; in the README’s words, “localized function names are not supported.” See [the workbook model](https://recursiveai.org/onedocument/docs/spreadsheets/), [rows of plain objects](https://recursiveai.org/onedocument/docs/spreadsheets-poco/) and [recalculation](https://recursiveai.org/onedocument/docs/spreadsheets-calc/). ## The onedocument tool ```shell dotnet tool install --global RecursiveAI.OneDocument.Tool --version 1.0.0 onedocument --help ``` Run in the folder the program wrote to: ```shell onedocument detect budget.xlsx onedocument extract report.docx onedocument convert report.docx report.odt onedocument render report.docx -o report.png --dpi 96 onedocument recalc budget.xlsx -o budget-recalc.xlsx ``` ```text Xlsx Site visit report Findings The roof drains are clear. The north stairwell light is out. Next steps Replace the stairwell light by Friday. wrote report.odt warning font-substitution: Font "Times New Roman" is not available; "Liberation Serif" has the same glyph widths and was used instead. 1 of 1 page(s) exported; wrote report.png 1 of 1 formulas calculated in 1 levels; 0 circular, 0 #SPILL! ``` The warning goes to standard error. `render` also writes PDF, with `--profile ua` or `--profile a2a` for PDF/UA-1 or PDF/A-2a, and takes `--pages` for a range. `inspect` lists what a file holds that a reader doesn’t see, and `sanitize` writes a copy without it. Here, the tracked change in `draft.docx`: ```shell onedocument inspect draft.docx onedocument sanitize draft.docx clean.docx onedocument extract clean.docx ``` ```text Document (Docx) revisions: 1 /word/document.xml: tracked changes [authors: Dana Reviewer] Docx -> Docx revisions: Removed, 1 removed, 1 before, 0 after removed /word/document.xml: tracked changes accepted Within 30 days, The supplier delivers the goods. ``` Sanitizing re-inspects what it wrote before it saves. It hasn’t been independently audited; the coverage review in the docs is the project’s own, and some categories are reported but not removed. See [inspect and sanitize](https://recursiveai.org/onedocument/docs/sanitize/) for every category and what is kept. ## Next - [Overview](https://recursiveai.org/onedocument/docs/overview/): every format, every check with its source, and the gaps. - [Coming from another library](https://recursiveai.org/onedocument/docs/migration/): the Open XML SDK, ClosedXML, EPPlus, NPOI, MiniExcel, ShapeCrawler and Xceed DocX, call by call. - [Templates](https://recursiveai.org/onedocument/docs/templating/): one template language for Word, Excel and PowerPoint files. - [Redaction](https://recursiveai.org/onedocument/docs/redaction/): removing text from the file, not only from view, with the output re-read to verify. # OneDocument **Read, write, calculate, template and render Word, Excel and PowerPoint documents (OOXML, legacy binary, RTF, ODF) in pure .NET. Not affiliated with Microsoft.** OneDocument is a clean-room document processor for `net10.0` that depends only on the .NET base class library and other Recursive AI libraries. It is NativeAOT-, trim- and browser-WebAssembly-friendly: no reflection, no native code, no `System.Xml` in the document path. - **Lossless models.** `Document`, `Workbook` and `Presentation` are views over the file's own markup. Parts you do not touch are written back byte-identical (raw zip entries, never recompressed); edits keep unknown elements, attributes, prefixes, `mc:AlternateContent` and `extLst` in order. - **Every layer is public:** containers and records, a lazy markup tree, generated typed views of the schemas, the hubs, and the services on top (calculation, templating, layout, operations). The `Office`, `Spreadsheet` and `Template` statics are one line over the model. - **Tolerant reading, typed defects.** Damage is repaired where Office repairs it and reported as a located defect (`docs/defects.md`); resource limits are on by default; nothing touches the network or runs code. - **Conversions say what they lose.** Every conversion between formats can return a loss report. ## Status: 1.0 The tables below say what the code does and what has been checked, with the note or test that holds each number. A check that could not be run says so and is listed under [Gaps](#gaps); nothing here claims what was not measured. Measurements were taken on the code at the 1.0 freeze. ## Samples Each sample under [`samples/`](https://github.com/recursive-ai-oss/onedocument-csharp/tree/a85720e075c783fc596641d8586ee126cae088c5/samples) is a small console program that generates its own input and runs in CI. | Task | Sample | The call | |---|---|---| | Convert a document, workbook or presentation to PDF | [`ConvertToPdf`](https://github.com/recursive-ai-oss/onedocument-csharp/tree/a85720e075c783fc596641d8586ee126cae088c5/samples/ConvertToPdf) | `Office.Convert("deck.pptx", "deck.pdf")` | | Fill a template and write PDF | [`TemplateToPdf`](https://github.com/recursive-ai-oss/onedocument-csharp/tree/a85720e075c783fc596641d8586ee126cae088c5/samples/TemplateToPdf) | `Template.Render("invoice.docx", invoice, "invoice.pdf")` | | Rows of plain objects to and from a workbook | [`SpreadsheetRows`](https://github.com/recursive-ai-oss/onedocument-csharp/tree/a85720e075c783fc596641d8586ee126cae088c5/samples/SpreadsheetRows) | `Spreadsheet.Write(path, orders)`, `Spreadsheet.Read(path)` | | Text of a legacy `.doc` | [`LegacyDocText`](https://github.com/recursive-ai-oss/onedocument-csharp/tree/a85720e075c783fc596641d8586ee126cae088c5/samples/LegacyDocText) | `Office.ExtractText("old.doc")` | | Edit a cell, recalculate, save as PDF | [`WorkbookRecalcPdf`](https://github.com/recursive-ai-oss/onedocument-csharp/tree/a85720e075c783fc596641d8586ee126cae088c5/samples/WorkbookRecalcPdf) | `Workbook.Open`, `cell.Value =`, `wb.Recalculate()`, `wb.SaveAs("x.pdf")` | | Filter paragraphs by style, accept tracked changes, save | [`DocumentRevisions`](https://github.com/recursive-ai-oss/onedocument-csharp/tree/a85720e075c783fc596641d8586ee126cae088c5/samples/DocumentRevisions) | `doc.Paragraphs.Where(p => p.Style == "Heading 1")`, `doc.Revisions.AcceptAll()`, `doc.SaveAs(...)` | | Sanitise in memory, then save | [`HappyPath`](https://github.com/recursive-ai-oss/onedocument-csharp/tree/a85720e075c783fc596641d8586ee126cae088c5/samples/HappyPath) | `doc.Sanitize()`, `doc.SaveAs("clean.docx")` | | Build a document, a workbook and a deck from nothing | [`Authoring`](https://github.com/recursive-ai-oss/onedocument-csharp/tree/a85720e075c783fc596641d8586ee126cae088c5/samples/Authoring) | `body.AddParagraph(...)`, `body.AddTable(rows)`, `sheet.Range("A1:C3").Values = ...`, `deck.AddSlide("Title Slide")` | [`HappyPath`](https://github.com/recursive-ai-oss/onedocument-csharp/tree/a85720e075c783fc596641d8586ee126cae088c5/samples/HappyPath) runs the introduction's sketch as written, one or two lines per task. ```csharp using RecursiveAI.OneDocument; using RecursiveAI.OneDocument.Documents; using var doc = Document.Open("contract.docx"); // the format comes from the content, never the extension doc.Paragraphs.First().SetText("Master services agreement"); doc.Save(); // atomic; only word/document.xml changes Office.Convert("contract.docx", "contract.pdf"); // needs the Rendering package (or the All metapackage) ``` The `Office` statics find the readers and writers of every kind package the application references; no registration call is needed (each package registers itself into the referencing C# project at build time, with no reflection). A format whose package is not referenced raises `FormatNotAvailableException` naming it. Referencing one kind package (say `.Spreadsheets`) is enough for that kind and pulls in no other. In a project that is not C# or that sets `false`, call `DocumentFormats.Register(FormatRegistry.Default)` (and `SpreadsheetFormats`, `PresentationFormats`, `RenderingFormats`) once at start-up; [`docs/contracts.md`](https://recursiveai.org/onedocument/docs/contracts/#formats-registry-and-sniffing-recursiveaionedocumentformats) has the table. Coming from another library? See [`docs/migration/`](https://github.com/recursive-ai-oss/onedocument-csharp/tree/a85720e075c783fc596641d8586ee126cae088c5/docs/migration). ## Formats "Untouched round trip" is open then save with no edit: for the OOXML, ODF and XLSB families every part is written back as stored; the other formats are rebuilt from the model and say so in the loss report. Details and defect codes are in the linked notes. | Format | Read | Write | Untouched round trip | Loss report | Notes | |---|---|---|---|---|---| | DOCX, DOCM, DOTX, DOTM, Flat OPC | yes | yes | parts byte-identical (`docs/contracts.md`, `DocumentOpenTests`) | n/a (native) | `docs/docx-*.md`, `docs/revisions.md` | | XLSX, XLSM, XLTX, XLTM, XLAM, Flat OPC | yes (model and streaming) | yes (model and streaming) | parts byte-identical (`docs/spreadsheets.md`) | n/a (native) | `docs/spreadsheets*.md` | | PPTX, PPTM, POTX, PPSX, Flat OPC | yes | yes | parts byte-identical (`docs/presentations.md`, `OpenAndSaveTests`) | n/a (native) | `docs/presentations.md` | | XLSB | yes (model and streaming) | yes | unedited parts copied byte for byte (`docs/xlsb.md`) | n/a (native) | `docs/xlsb.md` | | XLS (BIFF8) | yes (model and streaming) | yes (one clean BIFF8 dialect; RC4 CryptoAPI password optional) | rebuilt, not carried through | yes, on conversion | `docs/xls.md` | | XLS (BIFF2 to BIFF5) | yes (model and streaming) | no | n/a | yes | `docs/xls.md` (layouts written from observed files) | | DOC (Word 97 to 2013, Word 6/95, fast-saved) | yes | no (post-1.0 on demand) | n/a | yes | `docs/doc.md` | | PPT (97 to 2003) | yes | no (post-1.0 on demand) | n/a | yes | `docs/ppt.md` | | Word 2003 XML, Excel 2003 XML | yes | no | n/a | yes | `docs/xml-2003.md` | | RTF 1.9.1 | yes | yes | RTF saved without an edit keeps its bytes (`docs/rtf.md`) | yes, both directions | `docs/rtf.md` | | ODT, OTT, flat FODT | yes | yes | entries copied raw (`docs/odf.md`) | yes | `docs/odf.md` | | ODP, OTP, flat FODP | yes (`Presentation`) | yes | read and saved in its own format without an edit writes the original bytes; any edit writes a package of its own (`docs/odp.md`) | yes, both directions | `docs/odp.md` | | ODS, OTS, flat FODS | yes (`Workbook`) | yes | entries copied raw (`docs/odf.md`) | yes, both directions | `docs/ods.md` | | CSV, TSV | yes | yes | n/a | writing reports dropped formulas, styles and sheets | `docs/csv.md` | | HTML | yes (sanitised by default) | yes (documents; workbooks and presentations at a basic level) | n/a | yes | `docs/html.md` | | Markdown (CommonMark, GFM) | yes | yes | n/a | yes | `docs/markdown.md` | | PDF (tagged; PDF/UA-1 or PDF/A-2a profiles); PNG, JPEG, TIFF (multi-page), WebP and SVG pages | no | yes, for documents, workbooks and presentations | n/a | n/a | `docs/layout.md`, `docs/rendering-export.md` | | Text | no | yes (`Office.ExtractText`) | n/a | n/a | `docs/text-extraction.md` | | Embedded EMF, EMF+, WMF pictures | rendered | no | carried as parts | n/a | `docs/metafiles.md` | Encrypted files open when `OpenContext.Password` is supplied: OOXML Agile and Standard, RC4, RC4 CryptoAPI and XOR for the legacy formats (`docs/encryption.md`). Excel, Word and PowerPoint open our RC4 CryptoAPI XLS output, and the Agile and RC4 files other applications wrote, with the same password (9 of 10 fixtures; the XOR one is the exception). ## Checks Numbers come from the committed note or test named in the last column. Office-oracle checks (Word, Excel and PowerPoint opening our output without a repair prompt, Office's own result for the same operation, the layout differential) ran on desktop Office 16.0.20527 for the 1.0 gate: results, floors and the open items are in [`docs/office-oracle-1.0.md`](https://recursiveai.org/onedocument/docs/office-oracle-1-0/). The remaining requests under `testdata/` are listed there as not run. | Check | Result | Source | |---|---|---| | Untouched round trip over a public corpus, per-part hashes | 4,900 OOXML and XLSB packages (Common Crawl, bug-tracker attachments, LibreOffice-converted sets): 111,239 of 111,239 parts byte-identical at the package layer; 6 files refused with a typed defect. Through the hubs: 2,773 DOCX (56,316 parts), 1,508 XLSX (39,328), 537 PPTX (43,986) and 18 XLSB (334) saved, every part identical; an edit changes only the edited part in every file | `docs/gates-1.0.md` §1 | | Written OOXML against the ECMA-376 and extension schemas | no file went from valid to invalid by our write: DOCX 275 pairs, XLSX 360, PPTX 515 decks (2,101 outputs); legacy conversions to OOXML 301 of 301 valid and opened by LibreOffice 26.8; ODF 1.3 against the RelaxNG schema: ODT 59, ODS 51 and ODP 50 written files (results and open items in the note) | `docs/gates-1.0.md` §2, §3 | | Output opens in Word, Excel and PowerPoint with no repair prompt | 718 generated files open clean (edits, authoring, templates, mail merge, content controls, conversions from DOC, PPT, RTF, ODF, HTML, XLS, XLSB, written XLS with notes, layout cases); every other file is an input that fails the same way or an ODS file Excel does not open, each listed | `docs/office-oracle-1.0.md`, `testdata/*/ORACLE-REQUEST.md` | | Number formats against Excel (`TEXT` and cell display) | 7,602 of 7,602 agree on a grid; exceptions are listed | `docs/number-formats.md`, `NumberFormatExcelGridTests` | | Calculation: cached values reproduced over a corpus | recalculation reproduces the cached values of 99.52% of the testable formulas in 354 Common Crawl workbooks, 93.02% in 702 bug-tracker workbooks, 93.79% in 453 LibreOffice-converted workbooks, 95.14% in 450 XLS files, 99.54% in 18 XLSB files; every mismatch class is explained in the note. | `docs/gates-1.0.md` §5, `docs/calc.md`, `docs/spreadsheets-calc.md` | | Calculation: per-function grids against Excel 16.0.20527 | functions 22,700 rows 93.3 % agree, financial 84.5 %, engineering 88.0 %, lookup and reference 99.5 % (9,707 rows), database 100 % (324 rows); every disagreement is listed. | `docs/office-oracle-1.0.md`, `docs/calc.md`, `docs/spreadsheets-calc.md` | | Layout: page count, line breaks and object geometry against Office's PDF export (generated cases; rows where both sides had the same fonts) | Page count agrees on 196 of 196 documents and workbooks and 24 of 24 decks. Mean text-line agreement and, for Word, word positions within 2 pt: business documents 76 rows 0.951 and 0.872; long documents 15 rows 0.966 and 0.941; edge features (floats, drawings, diagrams, math) 29 rows 0.834 and 0.588; worksheets 26 rows 0.934 (geometry is information: Excel's default row height differs by display scaling); CJK and right-to-left 1 row 1.000 and 0.000 (8 more rows are font-substitution rows and do not gate); presentations 0.748 (geometry not run). These are floors: they only rise | `docs/office-oracle-1.0.md` §3, `docs/layout.md` | | Tagged PDF against veraPDF 1.30.3 (PDF/UA-1, PDF/A-2a) | PDF/UA-1: every PDF written passes (DOCX 11, XLSX 11, PPTX 16); a document the profile cannot make conformant is refused with the rule. PDF/A-2a: 19 of 20 DOCX and 15 of 16 PPTX pass; the failures are one rule, a private-use character without `ActualText` (a PDF-writer item). Generated layout set (224 cases, OnePdf 1.19.0; `tools/gate/pdf-sets.sh`): every file written passes both profiles. PDF/UA-1 wrote 200 (business 73 of 93, long 6 of 6, edge 59 of 63, CJK/RTL 7 of 7, decks 26 of 26, sheets 29 of 29); the other 24 are refused by our profile check with the rule (table headers, alt text). PDF/A-2a wrote 223 of 224 (one edge case refused). The set has no private-use characters. Footnotes are tagged `Div` in PDF/UA output until the PDF writer writes `/ID` on `Note` | `docs/gates-1.0.md` §6 | | Text extraction against Apache Tika 3.3.3 | 150 files per format: precision of our words in Tika's at least 0.985 for every format; the differences are what Tika emits and we leave out by policy | `docs/gates-1.0.md` §4 | | CommonMark and GFM conformance | 652 of 652 and 28 of 28 examples | `docs/markdown.md`, `MarkdownSpecTests` | | Metafiles against Windows' own pictures | 64 Windows-written metafiles compared (25 EMF, 24 EMF+, 15 WMF); per-case agreement in the note | `docs/metafiles.md`, `MetafileTests` | | Fuzzing, 600 s per parser | 70 targets, 600 s each under libFuzzer with SharpFuzz instrumentation, all clean on the final code. the campaign found and fixed crashes, hangs and limit failures (calculation loops and overflow, a template range clip, shared-string surrogates, R1C1 quoting, chart, drawing, layout and metafile clamps), each with a regression test | `docs/fuzzing.md`, `fuzz/` | | NativeAOT and trimming | the smoke app publishes with warnings as errors and runs | `smoke/aot`, `build/ci.sh aot` | | Browser WebAssembly | 31 smoke groups run in headless Chromium (Agile, Standard and RC4-MD5 decryption excepted, see Gaps) | `docs/platforms.md`, `build/ci.sh wasm-run` | | iOS and Android | the smoke checks pass in an iOS simulator and an Android emulator (Mono ahead-of-time release builds; NativeAOT on those hosts is not available for simulators) | `docs/platforms.md` | | Memory | 37 retained bytes per cell on a 20M-cell styled workbook (ClosedXML 79, NPOI 360); streaming read allocates 0.08 bytes per cell at 1M rows; streaming working set 47 to 49 MB from 10k to 1M rows (x64) | `bench/README.md` | | Speed (1.0 gate rows) | against the fastest like-for-like managed comparator over 49 rows, single thread: geomean 0.30 (x64) and 0.32 (arm64), worst row 1.08 (x64) and 1.07 (arm64); the save rows gate on the save-to-stream variants, and the save-to-file rows are published beside them (our `Save()` flushes the temp file to disk and renames it atomically, and the Open XML SDK edits the file in place without a flush); `.xls` and `.xlsb` reads go through forward cursors and are 0.55 to 0.64 times (xls) and 0.93 to 1.01 times (xlsb) the time of Sylvan.Data.Excel, the fastest forward reader | `bench/README.md`, `bench/published/` | ## Operations | Operation | State | |---|---| | Find and replace, merge, append, split, protection | implemented (`docs/operations-find-replace.md`, `docs/operations-merge.md`, `docs/protection.md`) | | Redaction (`Office.Redact`) | implemented, writes a new file and re-reads it to verify; the project's own completeness review (not an external audit) passed (`docs/redaction-review.md`), with its open items listed there. Office opens the redacted outputs: not run (see `docs/office-oracle-1.0.md`) | | Inspect and sanitise (`Office.Inspect`, `Office.Sanitize`, `onedocument inspect`, `sanitize`) | implemented (`docs/sanitize.md`); the Document Inspectors that automation can run (Word 5, Excel 5, PowerPoint 4) find nothing in 36 sanitised files that open, and Office opens them without repair (`docs/office-oracle-1.0.md`); the project's own coverage review (not an external audit) passed (`docs/sanitize-review.md`) | | VBA read, extract, remove | implemented, never executed (`docs/vba.md`) | | Accessibility checking | implemented (`docs/accessibility.md`); agreement with Office's checker is **pending**: the checker has no automation interface, so it was not measured (`docs/office-oracle-1.0.md`) | ## Gaps Stated here so they are not found by surprise. - **Encrypted output.** Encrypted files are written only for XLS (RC4 CryptoAPI). Agile OOXML and ODF encryption on write wait for the Cryptography package. - **Cryptography package.** `RecursiveAI.OneDocument.Cryptography` (signatures, encryption on write) has no content yet and is not part of the `All` metapackage. - **Browser.** WebAssembly has no `Aes` or MD5, so opening Agile or Standard encrypted OOXML and RC4-MD5 files there raises `OfficeUnsupportedException`. RC4 CryptoAPI and XOR files read normally (`docs/platforms.md`). The XOR reader's array-index rule rests on LibreOffice alone: Excel does not open our XOR test file, so that scheme is unconfirmed against Office (`docs/office-oracle-1.0.md`). - **Document Inspector agreement.** Office's object model exposes only some inspectors (Word 5, Excel 5, PowerPoint 4). Comments, revisions, document properties and personal information, embedded documents, macros and links cannot be asked, so agreement for those categories was not measured; our own re-inspection finds nothing left in sanitised output (`docs/office-oracle-1.0.md`, `docs/sanitize.md`). - **Accessibility Checker.** Pending: Office has no automation for it, so our results are checked against the published rule set only (`docs/accessibility.md`). - **PDF/A-2a.** In the veraPDF run, 19 of 20 DOCX and 15 of 16 PPTX pass; the failures are one rule (6.2.11.7.3, a private-use character without `ActualText`) in the PDF writer (`docs/gates-1.0.md` §6). - **Calculation.** Recalculation reproduces cached values over each corpus at the floors in the Checks table; the mismatches fall in named classes (precision beyond 9 digits, volatile functions and what depends on them, references into other workbooks, legacy references the converter leaves as `#REF!`, XLM macro functions, and unexplained cases), listed per set in `docs/gates-1.0.md` §5. ODS-sourced workbooks reproduce 65.3% and were not rerun after the fixes. The per-function grids against Excel disagree on 6.7% of function rows, 15.5% of financial and 12.0% of engineering rows (distribution inverses on boundary arguments, text conversions, coupon-date functions, iteration limits, error kinds on out-of-range arguments); `GETPIVOTDATA` over some cached pivots, `SUBTOTAL` with an autofilter and external references in BIFF are open. - **Schema validation.** On four already-invalid worksheets in the public corpus, the validator counts one more `xr:uid` error after our rewrite than before; the attribute is on the original too and its cause was not established (`docs/gates-1.0.md` §2). - **ODS.** No forward cursor: ODS reads through the model. - **Office opens ODS.** Excel does not open the ODS files we write (they validate against the ODF 1.3 schema and LibreOffice opens them); information, not bisected. - **Formula text** is en-US and invariant only; localized function names are not supported. - **DOC and PPT** are read-only. **Word 6/95 and BIFF2 to BIFF5** layouts come from observed files, not a specification. - **Metafiles.** Drawn as vector content where the interpreter supports the records; custom pen caps, line-end anchors, colour matrices and text clipping to a string's rectangle are not drawn, and some raster-operation cases fall back to a bitmap (`docs/metafiles.md`). - **Output formats.** PDF, and PNG, JPEG, TIFF, WebP and SVG pages (`PageExporter`; `SaveAs` and `Office.Convert` write the first page, or every page for TIFF). HTML, Markdown and text export are on the hubs. - **Rendering fidelity.** Page layout is implemented from the Word, PowerPoint and Excel rules in `docs/layout*.md`; how closely it matches Office is measured in `docs/office-oracle-1.0.md` (Word 16.0.20527, generated cases), and the layout notes list the places still to be checked against Word. - **Not executed.** Macros, OLE payloads, Power Query, the Data Model and Solver are carried and (for VBA) listed or removed, never run. ## Packages | Package | Contents | |---|---| | `RecursiveAI.OneDocument` | Core: contracts, containers (OPC, compound files, ODF package), XML tokenizer and markup tree, format registry, formula and calc core, number formats, template engine, text extraction, RTF and VBA layers | | `RecursiveAI.OneDocument.Documents` | Word-processing documents: DOCX, DOC, RTF, ODT, 2003 XML, HTML, Markdown, revisions, mail merge, operations | | `RecursiveAI.OneDocument.Spreadsheets` | Workbooks: XLSX, XLSB, XLS, CSV/TSV, 2003 XML, calculation, pivots, streaming, typed row binding | | `RecursiveAI.OneDocument.Presentations` | Presentations: PPTX, PPT | | `RecursiveAI.OneDocument.Schema` | Generated zero-copy typed views over every pinned schema: WordprocessingML, SpreadsheetML, PresentationML, DrawingML, charts, chartex, diagrams, VML, Office Math, the package parts and the Microsoft extension schemas; `.Spreadsheets` uses them | | `RecursiveAI.OneDocument.Rendering` | Layout, charts, SmartArt, equations, drawing; PDF and page-image (PNG, JPEG, TIFF, WebP, SVG) output for all three kinds | | `RecursiveAI.OneDocument.All` | Metapackage of every package above | | `RecursiveAI.OneDocument.Cryptography` | Signatures and encryption on write (no content yet; not in `All`) | | `RecursiveAI.OneDocument.Tool` | The `onedocument` command-line tool (`dotnet tool install -g RecursiveAI.OneDocument.Tool`): `detect`, `extract`, `recalc`, `template`, `render`, `convert`, `inspect`, `sanitize` | ## Building ``` dotnet build OneDocument.slnx dotnet test --solution OneDocument.slnx build/ci.sh # restore, build, test, samples, NativeAOT smoke, WebAssembly smoke, pack ``` The public surface of every library assembly is frozen in `src//PublicAPI.Shipped.txt`, checked on each build by the public-API analyzer (build-only, nothing flows into the packages): a public type or member added, removed or changed without a listing edit fails the build. List an intended change in `PublicAPI.Unshipped.txt` (`tools/api/update-public-api.py --unshipped` does it); a release moves those lines into `Shipped`. The .NET SDK version is pinned in `global.json`. Build output goes to `artifacts/` (ignored by git); set `ONEDOCUMENT_ARTIFACTS` to put it elsewhere. ## Layout `src/` packages, `samples/` runnable examples, `tests/` unit tests, `tools/` harness and generator tools, `fuzz/` libFuzzer targets, `bench/` benchmarks, `smoke/` NativeAOT and WebAssembly smoke apps, `docs/` normative notes (index in [`docs/README.md`](https://recursiveai.org/onedocument/docs/docs-map/)). ## Licence `MIT OR Apache-2.0`, © Recursive AI, LLC. See `LICENSE-MIT`, `LICENSE-APACHE` and `THIRD-PARTY-NOTICES.md`. OneDocument is not affiliated with Microsoft or with any other product or company named OneDocument. Word, Excel and PowerPoint are Microsoft product names used here only to describe the formats. # OneDocument notes Normative notes on how OneDocument behaves, one page per area. The code's XML documentation states the details of each type; these pages state the rules, the sources they come from and the deviations. Start with `contracts.md` and `defects.md`. ## Foundations | Note | What it covers | |---|---| | [contracts.md](https://recursiveai.org/onedocument/docs/contracts/) | Shared contracts: byte sources, limits, defects and exceptions, the format registry, the part model, the untouched-part round-trip rule | | [defects.md](https://recursiveai.org/onedocument/docs/defects/) | The defect catalogue: every `layer.problem` code, what it means and where it is located | | [packaging.md](https://recursiveai.org/onedocument/docs/packaging/) | OPC packages, Flat OPC and compound files under every format | | [markup.md](https://recursiveai.org/onedocument/docs/markup/) | The XML tokenizer, writer, lazy markup tree and reader | | [platforms.md](https://recursiveai.org/onedocument/docs/platforms/) | NativeAOT, trimming and browser WebAssembly: what is checked and the known browser gaps | | [encryption.md](https://recursiveai.org/onedocument/docs/encryption/) | Opening password-protected files; the supported schemes and what is written | | [protection.md](https://recursiveai.org/onedocument/docs/protection/) | Editing restrictions and their passwords for the three kinds | | [ole.md](https://recursiveai.org/onedocument/docs/ole/) | OLE property sets and embedded objects | | [vba.md](https://recursiveai.org/onedocument/docs/vba/) | VBA projects as data: read, list, extract, remove; never run | | [officeart.md](https://recursiveai.org/onedocument/docs/officeart/) | The OfficeArt reader the binary formats share | | [number-formats.md](https://recursiveai.org/onedocument/docs/number-formats/) | Excel's number-format language and date serials, as a shared service | | [text-extraction.md](https://recursiveai.org/onedocument/docs/text-extraction/) | Text extraction from OOXML packages | | [accessibility.md](https://recursiveai.org/onedocument/docs/accessibility/) | Accessibility checking against Office's published rules | ## Word-processing documents | Note | What it covers | |---|---| | [docx-styles-numbering.md](https://recursiveai.org/onedocument/docs/docx-styles-numbering/) | Style resolution and list labels | | [authoring.md](https://recursiveai.org/onedocument/docs/authoring/) | Building content with the hub: styles by name, runs, links, pictures, tables, ranges, slides; sanitising in memory | | [docx-tables-sections.md](https://recursiveai.org/onedocument/docs/docx-tables-sections/) | Tables, sections, headers and footers | | [docx-fields-notes-comments.md](https://recursiveai.org/onedocument/docs/docx-fields-notes-comments/) | Fields, footnotes and endnotes, comments | | [docx-content-controls.md](https://recursiveai.org/onedocument/docs/docx-content-controls/) | Content controls and custom XML binding | | [docx-stories-properties.md](https://recursiveai.org/onedocument/docs/docx-stories-properties/) | Text boxes, stories, list labels, document properties | | [revisions.md](https://recursiveai.org/onedocument/docs/revisions/) | Tracked changes: reading, authoring, accepting and rejecting | | [doc.md](https://recursiveai.org/onedocument/docs/doc/) | Reading Word 97 to 2013, Word 6/95 and fast-saved binary files | | [rtf.md](https://recursiveai.org/onedocument/docs/rtf/) | RTF 1.9.1 read and write, the group tree and the conversion | | [xml-2003.md](https://recursiveai.org/onedocument/docs/xml-2003/) | Word 2003 XML and Excel 2003 XML (read-only conversion) | | [html.md](https://recursiveai.org/onedocument/docs/html/) | HTML import (sanitised) and export | | [markdown.md](https://recursiveai.org/onedocument/docs/markdown/) | CommonMark and GFM import and export | | [odf.md](https://recursiveai.org/onedocument/docs/odf/) | OpenDocument package layer and text-document conversions | | [odp.md](https://recursiveai.org/onedocument/docs/odp/) | OpenDocument presentations (ODP, OTP, FODP) read and write | ## Workbooks | Note | What it covers | |---|---| | [spreadsheets.md](https://recursiveai.org/onedocument/docs/spreadsheets/) | The workbook hub and the cell store | | [spreadsheets-styles.md](https://recursiveai.org/onedocument/docs/spreadsheets-styles/) | Styles, tables, conditional formatting, validation, names | | [spreadsheets-streaming.md](https://recursiveai.org/onedocument/docs/spreadsheets-streaming/) | Forward-only row and cell cursors | | [spreadsheets-poco.md](https://recursiveai.org/onedocument/docs/spreadsheets-poco/) | `Spreadsheet.Read` and `Write`: rows of plain objects | | [spreadsheets-calc.md](https://recursiveai.org/onedocument/docs/spreadsheets-calc/) | Recalculation: dependency graph, levels, circular references, iteration, data tables, Goal Seek, external links | | [spreadsheets-pivots.md](https://recursiveai.org/onedocument/docs/spreadsheets-pivots/) | Pivot caches and tables, refresh | | [calc.md](https://recursiveai.org/onedocument/docs/calc/) | The formula tree, values, evaluator and function registry | | [calc-functions.md](https://recursiveai.org/onedocument/docs/calc-functions/) | The function families and their documented behaviour | | [csv.md](https://recursiveai.org/onedocument/docs/csv/) | CSV and TSV reader, writer and loss report | | [xlsb.md](https://recursiveai.org/onedocument/docs/xlsb/) | Binary workbooks | | [xls.md](https://recursiveai.org/onedocument/docs/xls/) | Legacy Excel workbooks, BIFF2 to BIFF8 read, BIFF8 write | ## Presentations | Note | What it covers | |---|---| | [presentations.md](https://recursiveai.org/onedocument/docs/presentations/) | The presentation hub, inheritance, editing | | [ppt.md](https://recursiveai.org/onedocument/docs/ppt/) | Reading PowerPoint 97-2003 files | ## Templating and operations | Note | What it covers | |---|---| | [templating.md](https://recursiveai.org/onedocument/docs/templating/) | The template syntax, extensions to formula syntax, data sources, the binder contract | | [templating-docx.md](https://recursiveai.org/onedocument/docs/templating-docx/) | The document template binder | | [templating-xlsx.md](https://recursiveai.org/onedocument/docs/templating-xlsx/) | The workbook template binder | | [templating-pptx.md](https://recursiveai.org/onedocument/docs/templating-pptx/) | The presentation template binder | | [mail-merge.md](https://recursiveai.org/onedocument/docs/mail-merge/) | Word mail merge | | [operations-merge.md](https://recursiveai.org/onedocument/docs/operations-merge/) | Merge, append, split and persisted field update | | [operations-find-replace.md](https://recursiveai.org/onedocument/docs/operations-find-replace/) | Find and replace across runs, cells and paragraphs | | [sanitize.md](https://recursiveai.org/onedocument/docs/sanitize/) | Inspecting a file for hidden data and personal information, and sanitising it | | [redaction.md](https://recursiveai.org/onedocument/docs/redaction/) | True redaction by match or region, and its verification | | [redaction-review.md](https://recursiveai.org/onedocument/docs/redaction-review/) | The project's own completeness review of redaction (not an external audit): method, defects found, verdict, open items | ## Rendering | Note | What it covers | |---|---| | [layout.md](https://recursiveai.org/onedocument/docs/layout/) | The layout engine and paint contract; PDF and PNG output | | [layout-tables.md](https://recursiveai.org/onedocument/docs/layout-tables/) | Tables in layout | | [layout-floats.md](https://recursiveai.org/onedocument/docs/layout-floats/) | Floating objects and text wrap | | [layout-fields.md](https://recursiveai.org/onedocument/docs/layout-fields/) | Fields that depend on pages; table of contents and indexes | | [spreadsheet-rendering.md](https://recursiveai.org/onedocument/docs/spreadsheet-rendering/) | Printing workbooks: pages, cell text, objects | | [presentation-rendering.md](https://recursiveai.org/onedocument/docs/presentation-rendering/) | Slides to PDF and PNG | | [rendering-export.md](https://recursiveai.org/onedocument/docs/rendering-export/) | Pages as PNG, JPEG, TIFF, WebP and SVG: page ranges, size, the SVG writer and its checks | | [drawingml-rendering.md](https://recursiveai.org/onedocument/docs/drawingml-rendering/) | Shapes, pictures, groups and effects | | [charts.md](https://recursiveai.org/onedocument/docs/charts/) | Chart reading, layout and painting | | [smartart.md](https://recursiveai.org/onedocument/docs/smartart/) | SmartArt: stored drawings and the layout engine | | [math-layout.md](https://recursiveai.org/onedocument/docs/math-layout/) | Equations (Office Math) | | [metafiles.md](https://recursiveai.org/onedocument/docs/metafiles/) | Drawing embedded EMF, EMF+ and WMF pictures | ## Coming from another library [migration/](https://recursiveai.org/onedocument/docs/migration/) maps the common tasks of seven libraries to OneDocument. # Shared contracts The contracts every layer of OneDocument is written against. They live in the core package (`RecursiveAI.OneDocument`). This note states the behaviour; the XML documentation on each type states the details. ## Sources (`RecursiveAI.OneDocument.IO`) `IByteSource` is input bytes however they are held. Reads are positioned (`Read(offset, buffer)`, `ReadAsync(offset, buffer, token)`), not tied to a cursor, so one source can serve several readers and a reader touches only the bytes it needs. A source never reads past its `Length`; a short read happens only at the end. | Source | Made by | Sync | Async | Zero-copy memory | |---|---|---|---|---| | memory | `ByteSource.FromMemory` | yes | yes | yes (`TryGetMemory`) | | file | `ByteSource.FromFile` | yes, concurrent | yes | no | | seekable stream | `ByteSource.FromStream` | yes, serialised | yes | no | | remote / ranged | `ByteSource.FromRanged` | no (`CanReadSync` is false) | yes | no | `ByteSource.ToMemoryAsync` buffers a remote source (bounded by a limit) for readers that need synchronous access. `ByteSource.AsStream` presents a source as a read-only seekable `Stream` for the stream-based archive readers. ## Limits (`OfficeLimits`) One immutable object, passed down through every layer inside `OpenContext`. All limits are on by default and must be positive. A limit that is hit raises `OfficeLimitException` naming the property (`LimitName`) and carrying a defect with the code `limit.`. Nothing is truncated silently. DTDs and entity declarations in XML are always refused; there is no limit to raise. | Property | Default | Why | |---|---|---| | `MaxTotalUncompressedSize` | 2 GiB | all parts together; `Large` raises it to 256 GiB | | `MaxExpansionRatio` | 200 : 1 | zip-bomb guard, checked after 32 MiB are produced; Office's own XML compresses 10-30 : 1 | | `MaxPartCount` | 50,000 | zip entries or compound-file streams | | `MaxPartSize` | 1 GiB | one part read whole into memory; streaming cursors are not bound by it | | `MaxXmlDepth` | 256 | open elements | | `MaxXmlAttributeCount` | 256 | per element, namespace declarations included | | `MaxXmlNameLength` | 1,024 | element, attribute and prefix names | | `MaxFormulaLength` | 8,192 | Excel's own limit | | `MaxFormulaDepth` | 64 | Excel's own limit | | `MaxRecursionDepth` | 256 | any other tree walk, group tree or evaluator | | `MaxRecordCount` | 100,000,000 | records in one binary stream | | `MaxRecordSize` | 256 MiB | one binary record | | `MaxAllocationBytes` | 1 GiB | any single allocation driven by input | | `MaxDefects` | 10,000 | defects a collection keeps (fatal ones always kept; the rest are counted) | `OfficeLimits.ToArchiveLimits()` maps total size, part count and expansion ratio onto the zip and compound-file readers; their path-length and depth limits keep the archive library's defaults. Parts of the library that call the archive readers translate its exceptions with the same names (`MaxEntryCount` surfaces as `MaxPartCount`). ## Defects and exceptions (`RecursiveAI.OneDocument.Diagnostics`, `OfficeException`) Reading is tolerant, so a repair is not an error, but it is never silent. A `Defect` has a closed `DefectKind` (where in the stack), a stable string `Code` (the precise problem; see `defects.md`), a `DefectSeverity` (`Repaired`, `Dropped`, `Fatal`), an English message for people, and a `DefectLocation`: - `Part`: part URI + byte offset in the uncompressed part; - `Record`: stream path + byte offset of the record; - `Cell`: sheet name + A1 address; - `Text`: character offset in the text (RTF, CSV, Markdown). Readers write defects to an `IDefectSink` (from any thread). `DefectCollection` keeps them in order up to a cap and counts the rest. **The contract the parsers honour:** malformed, hostile, truncated, encrypted or unsupported input surfaces as an `OfficeException` and never as an untyped exception (`IndexOutOfRangeException`, `InvalidCastException`, `OutOfMemoryException`, and so on). The subtypes are `OfficeFormatException`, `OfficeLimitException`, `OfficeEncryptedException` (password required or wrong), `OfficeUnsupportedException` and `FormatNotAvailableException` (no reader or writer registered). Each carries the fatal `Defect`. Caller mistakes (null arguments, disposed objects) and failures of the caller's own streams (`IOException`, `OperationCanceledException`) are the only other exceptions. Each format also offers `TryOpen`-style methods that return false instead of raising these. Code that calls the archive readers wraps every call and translates what it catches with the internal `ArchiveErrors.Translate`. ## Formats, registry and sniffing (`RecursiveAI.OneDocument.Formats`) `OfficeFormat` names every format; `OfficeFormats` gives each a descriptor (kind, media type, extensions, traits) and maps an extension to a format **for naming outputs only**. `FormatRegistry` is an explicit table of readers, text extractors and writers per format. Each kind's package registers what it implements. No application code is needed: a kind package ships a small C# file (`buildTransitive/`) that MSBuild compiles into the referencing project, where a module initializer calls the package's `Register(FormatRegistry.Default)`. It is plain source, not reflection or assembly scanning, so the trimmer and NativeAOT keep exactly the kinds the application references, and a project that references only core and `.Documents` has no spreadsheet code in its binary. Set `false` to leave it out (a project that fills a custom `FormatRegistry` itself, or a non-C# project, calls `DocumentFormats.Register` / `SpreadsheetFormats.Register` explicitly). Registering twice is harmless: a later registration replaces the earlier handler. `Office` and the document types use `FormatRegistry.Default`. For a kind whose package is not referenced, the operations raise `FormatNotAvailableException` naming the format. **How a consumer gets formats registered** | You have | What to do | |---|---| | A C# project that references one or more kind packages (`.Documents`, `.Spreadsheets`, `.Presentations`, `.Rendering`) or `.All` | Nothing. The package's build file compiles its registration into your project; the `Office` statics, `Document.Open`, `Workbook.Open` and the rest find the formats at once. A project that references only `.Spreadsheets` has spreadsheet formats and no word-processing code. | | A project that references only core | Nothing is registered. `Office.Open` on a format whose package is missing raises `FormatNotAvailableException` naming the format. | | `false`, a project that is not C# (F#, Visual Basic), or a build that compiles no package build files | Call the packages' `Register` once at start-up: `DocumentFormats.Register(FormatRegistry.Default)`, `SpreadsheetFormats.Register(...)`, `PresentationFormats.Register(...)`, `RenderingFormats.Register(...)`. Pass your own `FormatRegistry` instead of `Default` to keep a private set. | | A library that wraps OneDocument | Do not register on behalf of the application: reference the kind packages you need and let the application's build do it, or document the `Register` calls for it. Registering twice is harmless. | The consumer applications under `smoke/consumer` check each row that needs no call: one for each single kind package, the metapackage and a NativeAOT build, all with warnings as errors and no registration line. `FormatSniffer` identifies content, never an extension. It reads a 4,096-byte prefix and, for containers, the least more it needs: 1. zip signature: an entry named `mimetype` means OpenDocument and its text names the format; otherwise `[Content_Types].xml` plus the package relationship of type `.../officeDocument` name the main part, and the main part's content type names the format (Transitional and Strict spellings alike); a zip with neither is `Unknown`; 2. compound-file signature: the streams at the root (`EncryptedPackage` with `EncryptionInfo`, `WordDocument`, `Workbook`, `PowerPoint Document`, `Book`); 3. binary signatures: PDF, PNG, JPEG, TIFF, WebP, bare BIFF streams; 4. `{\rtf`; 5. XML by root element: Flat OPC (`package` in the `xmlPackage` namespace; the main part's content type, within 1 MiB, tells the kind), the 2003 formats, flat OpenDocument (`office:document` with `office:mimetype`), SVG, HTML; 6. last, delimited text: at least two complete lines of valid text with the same number of tabs, or of commas or semicolons, outside double quotes. Markdown and plain text are never detected. A damaged or unreadable container is `Unknown` (opening it reports the damage with its location). A resource limit being hit raises `OfficeLimitException`, so a bomb is not mistaken for junk. Remote sources are decided from the prefix when possible and buffered whole only for containers. ## Loss reports (`RecursiveAI.OneDocument.Diagnostics`) A format that is narrower than the model it is written from, and every conversion, says what it dropped. `ILossSink.Report(code, feature, count, message)` is the one method; `LossReport` is the collector (one `LossItem` per code, counts added, `IsLossless` when empty, `CountOf(code)`). A writer takes an optional `ILossSink`; null means the caller does not care. Codes are `loss.`, lower case, never reused, and are listed with the format that reports them (CSV: `loss.formulas`, `loss.cell-styles`, `loss.number-formats`, `loss.sheets`, `loss.merged-cells`, `loss.embedded-objects`; see `csv.md`; Word binary files: the `loss.doc-*` codes in `doc.md`). `feature` is words for people ("formulas"), `count` is the number of cells, sheets or paragraphs affected (at least 1), and `message` says what happened to them. Loss is not a defect: defects describe damage in the input, losses describe what the output cannot hold. ## Packages and parts (`RecursiveAI.OneDocument.Packaging`) `IPackage` is a set of named parts over a container; `IOpcPackage` adds relationships; the OPC and compound-file readers implement them. A part's bytes load lazily. **A part is untouched until someone replaces or opens it for writing**; reading never touches it. The writer copies untouched parts raw (a zip entry's compressed bytes), so they come out byte-identical. `PartUri` validates and compares OPC part names (ASCII case-insensitive; spelling kept) and resolves relationship targets. `Save()` replaces the file the package was opened from, atomically: the new content goes to a temporary file beside it (`....tmp`), is flushed to disk and renamed over the original (`AtomicFile`). A failed, cancelled or interrupted save leaves the original as it was. `InMemoryPackage` implements the contract in memory for tests and for generated documents; it is not a file format. ## Document base and the statics `OfficeFile` is the base of `Document`, `Workbook` and `Presentation`: `Format`, `Kind`, `Limits`, `FilePath`, `Defects`, `Save()`, `SaveAs(path)` (format chosen by extension), `SaveAs(path, format)`, `SaveAs(stream, format)`, `ExtractText`, `Dispose`. `SaveAs` writes a copy and never changes the object's own `Format` or `FilePath`. Documents are not safe for concurrent mutation; read-only use from many threads is safe after load. **Lazy parts in `Document`.** `Document.Open` and `FromPackage` read the package structure (content types, relationships) and nothing else. The main part is parsed the first time something needs it (`Body`, `Paragraphs`, `Markup`, `Revisions`, `IsStrict`, …), and styles, numbering, theme, notes and comments likewise. A main part that is not well-formed XML, or does not start with `w:document` (`docx.wrong-root`), therefore raises its `OfficeFormatException` at that first access, with the part name and offset, not at open; it raises again at every later access. `Save`, `SaveAs` and `ExtractText` on a document nobody has edited never parse the main part: it is copied as stored, so even a damaged main part survives a save unchanged. First-use initialisation is guarded, so concurrent readers share one tree and one hub object per part. `Office` holds the one-liners (`Detect`, `Open`, `ExtractText`, `Convert`). They take the same `OpenContext` (limits, defect sink, password) the layers below take, and add no options of their own. # Platforms OneDocument targets `net10.0` and runs the same code on the desktop runtime, under NativeAOT, in browser WebAssembly, in an iOS simulator and in an Android emulator. One set of smoke checks (`smoke/shared`) is compiled into the NativeAOT app (`smoke/aot`), the browser app (`smoke/wasm`), the iOS app (`smoke/ios`) and the Android app (`smoke/android`). ## What the smoke checks cover 31 groups: core contracts (sniffing, limits, registry, defects, packages), schema views, workbooks and recalculation, pivots, streaming, XLSB, CSV, XLS, Word binary, 2003 XML, POCO binding, templating, Markdown, VBA read and removal, accessibility, documents, presentations, rendering (layout, drawing, SmartArt, floats) to PDF and PNG with the bundled substitute fonts, number formats, calc, text extraction, and a `platform` group for what differs between hosts: - regular-expression time limits (a catastrophic pattern stops at its limit); - culture and encodings (output never depends on the host culture; UTF-8, UTF-16 and Latin-1 only); - parallel recalculation with the thread count forced above one, then an incremental pass; - files through streams: save to a path, the atomic `Save()` (temp file and rename), reopen, CSV from a path; - one password-protected file per encryption scheme (agile DOCX and XLSX, RC4 Word and Excel, XOR, RC4 CryptoAPI). ## Running the browser suite ``` build/ci.sh wasm-run ``` It needs the `wasm-tools` workload (`dotnet workload install wasm-tools`), `python3`, `curl` and `unzip`. The step: 1. publishes `smoke/wasm` with warnings as errors, so a trim or AOT warning fails it; 2. fetches the pinned Chrome for Testing build into `$ONEDOCUMENT_HOME/oracles/chromium-/` (default `~/.onedocument`), checked against its SHA-256 (`build/wasm/fetch-chromium.sh`); 3. serves the app on 127.0.0.1 and opens it in headless Chromium, which is told to resolve nothing else, so the run uses no network (`build/wasm/run-browser.py`); 4. collects every console line the app prints and exits non-zero unless the app exits 0 and prints `SMOKE-RESULT ok`. The log is `$CI_OUT/wasm-run.log`: `RUNTIME`, `CRYPTO` (the probe below), `GROUP` (one line per group with its time), `FAIL`, `ENCRYPTED` (what each protected file did) and the final `SMOKE-RESULT`. The browser app runs on the Mono interpreter (no ahead-of-time compilation) on one thread; the whole run takes about five seconds. ## Running on iOS (simulator) ``` build/ci.sh ios-run ``` The hosted CI job runs it only when the runner image has the Xcode the `ios` workload requires, and otherwise skips it with a notice; the 1.0 iOS results come from a Mac with Xcode 27.1 and the iOS 27.1 simulator. Needs macOS with Xcode, an iOS simulator runtime and the `ios` workload (`dotnet workload install ios`; a user-local SDK works). `build/mobile/ios-run.sh` builds `smoke/ios` for the simulator with warnings as errors, installs it on a simulator (an existing device is reused and shut down afterwards; otherwise one is created and deleted), launches it and reads the app's console. The log is `$CI_OUT/ios-run.log`, with the same line types as the browser log; the exit code is the result (`SMOKE-RESULT ok`). What is built is the SDK's release app for the simulator: trimmed, and every assembly ahead-of-time compiled by the Mono compiler. NativeAOT on iOS is for devices only (the SDK rejects a simulator runtime identifier with NETSDK1203), so the NativeAOT compiler cannot be exercised in a simulator; the NativeAOT smoke app on the desktop covers that compiler with the same checks. The app runs the checks before any UI exists and exits: current iOS releases stop an app that starts UIKit without scene lifecycle support, and the checks need no screen. The iOS SDK writes code-signing stamp files at a path built from the relative path between the project and the app bundle; with the output directory outside the repository that path can land on a file inside the bundle and overwrite it, so the script keeps its output under `artifacts/ios` in the repository (git ignores it). ## Running on Android (emulator) ``` JAVA_HOME=/path/to/jdk ANDROID_SDK_ROOT=/path/to/android-sdk build/ci.sh android-run ``` Needs Linux with `/dev/kvm` access, a JDK 17 or newer (`JAVA_HOME`), the `android` workload, and an Android SDK whose licence you have already accepted, with the emulator, platform-tools, a platform, build tools and an x86_64 system image. The script installs nothing and never changes that SDK: it creates its own virtual device in `ANDROID_AVD_HOME` (default `$ONEDOCUMENT_HOME/android-avd`) from the newest installed x86_64 image. `build/mobile/android-run.sh` publishes `smoke/android` (warnings as errors), starts a headless emulator on a fixed port, installs the app, launches it and reads the `OneDocumentSmoke` tag from `logcat`. The log is `$CI_OUT/android-run.log`. The app is the SDK's trimmed release build with Mono; NativeAOT on Android is experimental and the SDK refuses it (XA1040). The run is not part of the hosted CI workflow: it runs on a build machine with KVM. ## Known browser gaps The browser runtime has one thread (`Environment.ProcessorCount` is 1). Recalculation and layout run serially there; forcing the parallel path still gives the same numbers. Blocking on a pool task from the main thread throws, so the library never does it on these paths. Cryptography in the browser runtime (probed by `PlatformSmoke.CryptoProbe`): | Primitive | Browser | |---|---| | SHA-1, SHA-256, SHA-384, SHA-512 (`HashData`, `IncrementalHash`) | works | | HMAC over SHA-1/256/384/512 (`HMACSHA*`, `IncrementalHash.CreateHMAC`) | works | | `RandomNumberGenerator`, `CryptographicOperations.FixedTimeEquals` | works | | MD5 (`MD5.HashData`, `IncrementalHash`, HMAC over MD5) | `CryptographicException` | | `Aes` ECB, CBC and CFB | `PlatformNotSupportedException` | The library turns both gaps into `OfficeUnsupportedException` that names the missing primitive. Consequences in the browser: - reading agile or Standard encrypted OOXML (needs AES) and RC4 Word and Excel files with the MD5-derived key (needs MD5) raises `OfficeUnsupportedException`; - RC4 CryptoAPI (SHA-1) and XOR obfuscation read normally, and the XLS RC4 CryptoAPI writer works (SHA-1 only); - unencrypted files are unaffected. This is a known limitation of 1.0: the library does not carry its own AES or MD5, and opening these encrypted files in the browser raises the typed error above. It will be revisited if users ask for it. The browser build uses invariant globalization, as every build of the library does; nothing depends on ICU. ## Size The published browser app (interpreter build, trimmed) is about 62 MB on disk including the precompressed copies the SDK writes next to each file; the .NET assemblies and fonts are 35 MB uncompressed and 11.7 MiB when served Brotli-compressed. The bundled substitute fonts are the largest part. # Coming from another library These pages map the common tasks of seven libraries to OneDocument, call by call. They are written from each library's published API documentation. OneDocument has no compatibility layer and no shim: the pages are a map, and the code on the right is ordinary OneDocument. | You use | Page | |---|---| | EPPlus 4.x | [epplus.md](https://recursiveai.org/onedocument/docs/migration-epplus/) | | ClosedXML | [closedxml.md](https://recursiveai.org/onedocument/docs/migration-closedxml/) | | NPOI (XSSF, HSSF, XWPF) | [npoi.md](https://recursiveai.org/onedocument/docs/migration-npoi/) | | Open XML SDK | [open-xml-sdk.md](https://recursiveai.org/onedocument/docs/migration-open-xml-sdk/) | | Xceed DocX | [xceed-docx.md](https://recursiveai.org/onedocument/docs/migration-xceed-docx/) | | MiniExcel | [miniexcel.md](https://recursiveai.org/onedocument/docs/migration-miniexcel/) | | ShapeCrawler | [shapecrawler.md](https://recursiveai.org/onedocument/docs/migration-shapecrawler/) | ## Things that are the same on every page - **Packages.** Reference `RecursiveAI.OneDocument.All`, or only the kind you need: `.Spreadsheets` for workbooks, `.Documents` for word-processing documents, `.Presentations` for presentations, `.Rendering` for PDF and PNG output. No registration call is needed. - **Opening finds the format from the content.** `Workbook.Open`, `Document.Open` and `Presentation.Open` take a path, a stream or a byte source; the extension is never trusted. `Office.Open` returns whichever kind the file is. - **Opening never recalculates.** Cached values are what you read. `Workbook.Recalculate()` runs the formulas when you ask. - **Unknown content survives.** A part you did not touch is written back byte-identical; an edited part keeps the elements and attributes the model does not know. - **Cells are addressed on the sheet**, `sheet["B2"]` or `sheet[row, column]` (1-based). There is no range object with bulk setters; loop, or use the typed-row binding for tables of data. - **Damage is a defect, not a crash.** What Office repairs on open is repaired and listed in `file.Defects` with a location; hostile input hits limits (`OfficeLimits`) and raises a typed exception. - **No network.** External links, remote templates and linked pictures are default-deny and go through resolvers you supply. - **No speed claims here.** Benchmark rows against these libraries are published with the 1.0 gate (`bench/README.md` has the preview rows and the method). Where a task has no high-level OneDocument call yet, the page says so and names the lower layer that does it (the typed views, `.View`, over the same markup, or the markup tree). # Coming from the Open XML SDK The Open XML SDK gives you the packages, parts and a typed class per schema element: `WordprocessingDocument`, `SpreadsheetDocument`, `PresentationDocument`, `OpenXmlElement` and its subclasses, `OpenXmlReader` / `OpenXmlWriter` for streaming. OneDocument has two ways to work at that level, and a hub above it. Read [README.md](https://recursiveai.org/onedocument/docs/migration/) first for what is true on every page. | SDK idea | OneDocument | |---|---| | `WordprocessingDocument.Open(path, true)` | `Document.Open(path)` | | `SpreadsheetDocument.Open(path, true)` | `Workbook.Open(path)` | | `PresentationDocument.Open(path, true)` | `Presentation.Open(path)` | | The package, parts, relationships | `doc.Package` (`IOpcPackage`): parts by URI, relationships, content types (`docs/packaging.md`) | | Typed element classes (`Paragraph`, `Run`, `Worksheet`, ...) | Generated typed views in `RecursiveAI.OneDocument.Schema.*`, reached with `.View` on a hub object | | `OpenXmlElement` tree | The markup tree (`MarkupNode`, `MarkupRef`): lazy, over the part buffer, lossless (`docs/markup.md`) | | `OpenXmlReader` / `OpenXmlWriter` | `XlsxStreamReader` / `XlsxStreamWriter` for sheets; the XML tokenizer and writer for anything else | | `OpenXmlValidator` | not provided; written files are checked against the pinned schemas in the repository's own harness | ## Open, read text, edit, save ```csharp // Open XML SDK using var doc = WordprocessingDocument.Open(path, true); Body body = doc.MainDocumentPart!.Document.Body!; foreach (Paragraph p in body.Descendants()) { Console.WriteLine(p.InnerText); } body.AppendChild(new Paragraph(new Run(new Text("Added")))); doc.MainDocumentPart.Document.Save(); ``` ```csharp // OneDocument using Document doc = Document.Open(path); foreach (Paragraph p in doc.Paragraphs) { Console.WriteLine(p.Text); } doc.Body.AddParagraph("Added"); doc.Save(); ``` The hub gives the common reads and edits (paragraphs, runs, tables, styles, numbering, sections, headers and footers, fields, notes, comments, revisions, content controls). When it does not have a call, drop to the typed view of the same element: ```csharp // OneDocument: a typed view over the same markup, no conversion var view = doc.Paragraphs.First().View; // RecursiveAI.OneDocument.Schema.Wordprocessing.Paragraph ``` Differences that matter when moving code: - **No DOM load.** The markup tree is lazy: a part is parsed when first touched, and a part nobody edited is written back as stored. `Save()` is an atomic replace. - **No `Dispose` of parts.** The document owns everything; dispose the document. - **Unknown content is kept** by construction (prefixes, `mc:AlternateContent`, `extLst`, order); you do not have to model it. - **Typed views are zero-copy wrappers**, not a separate tree: editing through a view edits the markup the hub reads. - **No `OpenXmlCompositeElement.Append` family of validation.** Write what the schema allows; the typed views create children in schema order (`AddX`, `GetOrAddX`). ## Spreadsheets ```csharp // Open XML SDK using var ss = SpreadsheetDocument.Open(path, true); WorkbookPart wbp = ss.WorkbookPart!; Sheet sheet = wbp.Workbook.Descendants().First(s => s.Name == "Sales"); var wsp = (WorksheetPart)wbp.GetPartById(sheet.Id!); Cell cell = wsp.Worksheet.Descendants().First(c => c.CellReference == "B2"); // ...resolve shared strings and styles yourself ``` ```csharp // OneDocument using Workbook wb = Workbook.Open(path); string text = wb.Sheets["Sales"]["B2"].Text; // shared strings, styles and dates resolved double n = wb.Sheets["Sales"]["B2"].Number; ``` ### Streaming a large sheet ```csharp // Open XML SDK using OpenXmlReader reader = OpenXmlReader.Create(worksheetPart); while (reader.Read()) { if (reader.ElementType == typeof(Cell) && reader.IsStartElement) { /* ... */ } } ``` ```csharp // OneDocument using XlsxStreamReader book = XlsxStreamReader.Open(path); using SheetCursor sheet = book.OpenSheet(0); while (sheet.ReadRow()) { while (sheet.ReadCell()) { /* sheet.Kind, sheet.Number, sheet.TextUtf8 */ } } ``` Shared strings, styles and date detection are resolved by the cursor; it allocates nothing per cell (`docs/spreadsheets-streaming.md`). ## Presentations ```csharp // Open XML SDK using var ppt = PresentationDocument.Open(path, true); SlideIdList ids = ppt.PresentationPart!.Presentation.SlideIdList!; ``` ```csharp // OneDocument using Presentation deck = Presentation.Open(path); foreach (Slide slide in deck.Slides) { Console.WriteLine(slide.Title); } ``` ## Not supported or different - No validator class. Output validation against the ECMA-376 and extension schemas is a repository test harness, not an API. - No `OpenXmlPackage.SaveAs` with package-level edits through a part collection API beyond `IOpcPackage` (parts are added and replaced by URI). - Typed views are generated for the schemas OneDocument pins (ECMA-376 and the Microsoft extension schemas); a namespace outside them stays reachable as markup. - No `FlatOpc` helper class: Flat OPC is a save format (`doc.SaveAs("x.xml")`) and is read by content. # Coming from ClosedXML ClosedXML edits workbooks through `XLWorkbook`, `IXLWorksheet`, `IXLCell` and `IXLRange`; ClosedXML.Report fills templates. OneDocument's equivalents are `Workbook`, `Worksheet`, `Cell`, `Spreadsheet` and `Template`. Read [README.md](https://recursiveai.org/onedocument/docs/migration/) first for what is true on every page. ## Open, read, write, save ```csharp // ClosedXML using var wb = new XLWorkbook(path); IXLWorksheet ws = wb.Worksheet("Sales"); ws.Cell("B2").Value = 1200; ws.Cell("A2").Value = "North"; double d = ws.Cell("B2").GetDouble(); string s = ws.Cell("A2").GetString(); wb.Save(); ``` ```csharp // OneDocument using Workbook wb = Workbook.Open(path); Worksheet ws = wb.Sheets["Sales"]; ws["B2"].Value = 1200; ws["A2"].SetString("North"); double d = ws["B2"].Number; string s = ws["A2"].Text; wb.Save(); ``` `ws.Cell(row, column)` is `ws[row, column]`. A new workbook is `Workbook.Create()`; `ws.Cell(...).Value = "text"` is `SetString` because a string cell holds a handle into the workbook's string table. `SaveAs(path)` chooses the format by extension (`.xlsx`, `.xlsb`, `.xls`, `.csv`, `.pdf`, ...). ## Formulas ```csharp // ClosedXML ws.Cell("B5").FormulaA1 = "SUM(B2:B4)"; wb.RecalculateAllFormulas(); ``` ```csharp // OneDocument ws["B5"].Formula = "SUM(B2:B4)"; wb.RecalculateAll(); // or wb.Recalculate() for only what changed ``` Reading a formula cell returns its **cached** value; ClosedXML evaluates on demand, OneDocument never does until you call `Recalculate`. `FormulaR1C1` has a counterpart in the public formula tree (`docs/calc.md`). ## Styles ```csharp // ClosedXML ws.Range("A1:C1").Style.Font.Bold = true; ws.Cell("B2").Style.NumberFormat.Format = "#,##0.00"; ws.Range("A1:C1").Merge(); ``` ```csharp // OneDocument ws.Range("A1:C1").UpdateStyle(s => s with { Font = (s.Font ?? new CellFont()) with { Bold = true } }); ws.Range("B2").NumberFormat = "#,##0.00"; ws.Range("A1:C1").Merge(); ``` A `SheetRange` is a handle over the sheet, not a style object: styles are immutable records, and `UpdateStyle` applies the change to every cell of the range (once per distinct current style). ## Tables and data ```csharp // ClosedXML ws.Cell("A1").InsertTable(orders); ws.Range("A1:D10").CreateTable("Orders"); ``` ```csharp // OneDocument Spreadsheet.Write(path, orders); // [SpreadsheetRow] on the type; also .csv and .tsv ws.Tables.Add("Orders", CellRange.Parse("A1:D10")); // a table over cells already there ``` `Spreadsheet.Read(path)` is the typed way back (`docs/spreadsheets-poco.md`). There is no call that inserts a collection at a cell of an open workbook. ## Templates (ClosedXML.Report) ```csharp // ClosedXML.Report var template = new XLTemplate(path); template.AddVariable(invoice); template.Generate(); template.SaveAs(outPath); ``` ```csharp // OneDocument Template.Render(path, invoice, outPath); // the same call for .xlsx, .docx and .pptx ``` The tag syntax differs: OneDocument uses `{{ expression }}` and `{% for item in items %}` with Excel formulas as expressions (`docs/templating.md`, `docs/templating-xlsx.md`). Existing ClosedXML.Report templates must be rewritten. ## PDF ClosedXML does not render. `Office.Convert("in.xlsx", "out.pdf")` does, with the Rendering package. ## Streaming ClosedXML loads the whole workbook. For large sheets use `XlsxStreamReader` and `XlsxStreamWriter` (`docs/spreadsheets-streaming.md`). ## Not supported or different - Ranges (`ws.Range("A1:C10")`) read and write values as a two-dimensional array and set styles, number formats, merges and `Clear`; there are no `IXLRows` / `IXLColumns` objects, and no copy or sort on a range. - No `AdjustToContents`, `InsertRowsAbove` style layout helpers; column width and row height are set through `ColumnProperties` and `GetRow(row).Height`. - `ws.AddPicture(bytes, "E1", widthPixels, heightPixels, altText)` adds a picture; there is no chart or shape creation call. Existing ones are kept and rendered. - Formulas are not evaluated on read; no `ws.Evaluate("...")` shortcut on the sheet. - Pivot tables: existing ones are kept and can be refreshed (`docs/spreadsheets-pivots.md`); there is no pivot builder. # Coming from EPPlus 4.x EPPlus 4.x edits workbooks through `ExcelPackage`, `ExcelWorkbook`, `ExcelWorksheet` and `ExcelRange`. OneDocument's workbook is `Workbook` / `Worksheet` / `Cell`. (Later EPPlus versions have a different licence and are not covered.) Read [README.md](https://recursiveai.org/onedocument/docs/migration/) first for what is true on every page. ## Open, read, write, save ```csharp // EPPlus 4.x using var package = new ExcelPackage(new FileInfo(path)); ExcelWorksheet ws = package.Workbook.Worksheets["Sales"]; object v = ws.Cells["B2"].Value; ws.Cells["B2"].Value = 1200; ws.Cells["A2"].Value = "North"; package.Save(); ``` ```csharp // OneDocument using Workbook wb = Workbook.Open(path); Worksheet ws = wb.Sheets["Sales"]; CellValue v = ws["B2"].Value; // .Text, .Number, .FormattedText read it other ways ws["B2"].Value = 1200; ws["A2"].SetString("North"); wb.Save(); // atomic replace; only the edited sheet part changes ``` A new workbook: `ExcelPackage()` + `Worksheets.Add("Sales")` becomes `Workbook.Create()` + `wb.Sheets.Add("Sales")`. Save to a stream with `wb.SaveAs(stream, OfficeFormat.Xlsx)`; `SaveAs(path)` picks the format from the extension. ## Formulas and calculation ```csharp // EPPlus 4.x ws.Cells["B5"].Formula = "SUM(B2:B4)"; package.Workbook.Calculate(); ``` ```csharp // OneDocument ws["B5"].Formula = "SUM(B2:B4)"; // no leading '=' wb.Recalculate(); // incremental; wb.RecalculateAll() for everything ``` The calculation engine is OneDocument's own (spill, `LET`, `LAMBDA`, structured references). Settings such as iteration are on the workbook (`SetIterativeCalculation`). `docs/spreadsheets-calc.md` lists what is calculated and the documented deviations. ## Styles and number formats ```csharp // EPPlus 4.x ws.Cells["A1"].Style.Font.Bold = true; ws.Cells["B2"].Style.Numberformat.Format = "#,##0.00"; ``` ```csharp // OneDocument ws.Range("A1").UpdateStyle(s => s with { Font = (s.Font ?? new CellFont()) with { Bold = true } }); ws.Range("B2").NumberFormat = "#,##0.00"; ``` Styles are immutable records (`CellStyleSpec`); `UpdateStyle` returns a new one, for a cell (`ws["A1"].UpdateStyle`) or a block (`ws.Range("A1:C3").UpdateStyle`). `ws["A1"].FormattedText` gives the text as Excel displays it. ## Tables, names, merges, dates ```csharp // EPPlus 4.x ws.Tables.Add(ws.Cells["A1:D10"], "Orders"); ws.Cells["A1:C1"].Merge = true; ws.Cells["C2"].Value = new DateTime(2026, 3, 1); ``` ```csharp // OneDocument ws.Tables.Add("Orders", CellRange.Parse("A1:D10")); ws.Range("A1:C1").Merge(); ws["C2"].SetDate(new DateTime(2026, 3, 1)); wb.Names.Add("Rate", "Sales!$B$1"); ``` Conditional formatting and data validation are `ws.ConditionalFormats` and `ws.Validations` (`docs/spreadsheets-styles.md`). ## Loading collections ```csharp // EPPlus 4.x ws.Cells["A1"].LoadFromCollection(orders, true); ``` ```csharp // OneDocument: mark the type [SpreadsheetRow] (a partial class or record), then Spreadsheet.Write(path, orders); // or .csv, .tsv foreach (Order o in Spreadsheet.Read(path)) { } ``` The binding is generated at compile time (no reflection) and streams: see `docs/spreadsheets-poco.md`. To write the same data into a sheet of an open workbook, loop and set cells. ## Streaming EPPlus 4.x has no streaming reader or writer. OneDocument has `XlsxStreamReader` / `XlsxStreamWriter` (constant memory, no allocation per cell on read): `docs/spreadsheets-streaming.md`. ## PDF EPPlus 4.x does not render. `wb.SaveAs("report.pdf")` or `Office.Convert("in.xlsx", "out.pdf")` does, with the Rendering package. ## Not supported or different - `ws.Range("A1:C3")` is a handle with `Values` (a two-dimensional array), `UpdateStyle`, `NumberFormat`, `Clear` and `Merge`; there is no `Copy`, `AutoFitColumns` or `LoadFromCollection` on it (use `Spreadsheet.Write` for rows of objects). - `ws.AddPicture(bytes, "E1", ...)` adds a picture. Charts are not created through a high-level call; existing ones are kept and rendered. - No sparkline, form control or shape creation API. - Password protection: sheet and workbook protection are supported (`docs/protection.md`); encrypted files are written only as XLS (RC4) for now. - Cell values are `CellValue` (kind plus payload), not `object`; strings are handles into the workbook's string table, hence `SetString`. - Rows and columns are 1-based on both sides; `ws[row, column]` mirrors `Cells[row, column]`. - Templates: EPPlus 4.x has none; see `docs/templating-xlsx.md`. # Coming from NPOI NPOI has three families that matter here: `XSSF*` (OOXML workbooks), `HSSF*` (legacy `.xls`) and `XWPF*` (Word documents). OneDocument has one model per kind (`Workbook`, `Document`, `Presentation`) that opens whichever file format holds it. Read [README.md](https://recursiveai.org/onedocument/docs/migration/) first for what is true on every page. ## Workbooks: open, read, write, save ```csharp // NPOI using var fs = File.OpenRead(path); IWorkbook wb = new XSSFWorkbook(fs); // HSSFWorkbook for .xls, or WorkbookFactory.Create ISheet sheet = wb.GetSheet("Sales"); IRow row = sheet.GetRow(1) ?? sheet.CreateRow(1); ICell cell = row.GetCell(1) ?? row.CreateCell(1); cell.SetCellValue(1200); using var outFs = File.Create(outPath); wb.Write(outFs); ``` ```csharp // OneDocument using Workbook wb = Workbook.Open(path); // .xlsx, .xlsb and .xls all open as a Workbook Worksheet ws = wb.Sheets["Sales"]; ws[2, 2].Value = 1200; // NPOI rows and columns are 0-based; here they are 1-based wb.SaveAs(outPath); // .xlsx, .xlsb, .xls, .csv or .pdf by extension ``` No row or cell creation step: assigning a value creates the cell. Row and column numbers are **1-based**. ## Formulas and evaluation ```csharp // NPOI cell.SetCellFormula("SUM(B2:B4)"); wb.GetCreationHelper().CreateFormulaEvaluator().EvaluateAll(); ``` ```csharp // OneDocument ws["B5"].Formula = "SUM(B2:B4)"; wb.RecalculateAll(); ``` ## Styles ```csharp // NPOI ICellStyle style = wb.CreateCellStyle(); style.DataFormat = wb.CreateDataFormat().GetFormat("#,##0.00"); cell.CellStyle = style; ``` ```csharp // OneDocument ws["B2"].UpdateStyle(s => s with { NumberFormatCode = "#,##0.00" }); ``` The workbook's style table is managed for you; a style is a record, not a shared object to create and attach. ## Legacy .xls OneDocument reads BIFF2 to BIFF8 and writes one BIFF8 dialect (`wb.SaveAs("out.xls")`, or with a password for RC4 encryption). HSSF's low-level record classes have no counterpart; the record layer is `RecursiveAI.OneDocument.Spreadsheets.Biff` (`docs/xls.md`). ## Word documents (XWPF) ```csharp // NPOI using var fs = File.OpenRead(path); var doc = new XWPFDocument(fs); foreach (XWPFParagraph p in doc.Paragraphs) { Console.WriteLine(p.Text); } XWPFParagraph np = doc.CreateParagraph(); np.CreateRun().SetText("Added"); using var outFs = File.Create(outPath); doc.Write(outFs); ``` ```csharp // OneDocument using Document doc = Document.Open(path); // also .doc, .rtf, .odt foreach (Paragraph p in doc.Paragraphs) { Console.WriteLine(p.Text); } doc.Body.AddParagraph("Added"); doc.SaveAs(outPath); ``` Paragraph style is `p.Style` (the name Word shows, "Heading 1") or `p.StyleId`; `Body.AddParagraph(text)` adds a paragraph. `doc.Body.AddTable(rows)` creates a table and `doc.Tables` edits cells and rows (`docs/docx-tables-sections.md`). A run is formatted with `run.Bold`, `run.Italic`, `run.Underline`, `run.FontSize`, `run.Color` and `run.FontName`; the typed view `run.View` reaches everything else (`docs/authoring.md`). ## Text extraction ```csharp // NPOI (the extractor classes) var extractor = new XWPFWordExtractor(doc); string text = extractor.Text; ``` ```csharp // OneDocument string text = Office.ExtractText(path); // any supported format, streaming where it can ``` ## Streaming NPOI's `SXSSFWorkbook` streams writes. OneDocument's `XlsxStreamWriter` and `XlsxStreamReader` stream both ways (`docs/spreadsheets-streaming.md`). ## Not supported or different - No `HSLF`/`XSLF` (PowerPoint) object model matching: use `Presentation` (`docs/presentations.md`); `.ppt` is read-only. - No writing of `.doc` or `.ppt`. - Pictures are added with `body.AddPicture(bytes, widthPoints, ...)`, `sheet.AddPicture(bytes, "E1")` and `slide.AddPicture(bytes, left, top, width)`; there is no drawing-shape or chart creation call (existing ones are kept and rendered). - No `IRow` / `IFormulaEvaluator` objects; no `ICellStyle` instances. - `.xls` support is BIFF8 for writing; the formula evaluation engine is not NPOI's, and results are documented in `docs/spreadsheets-calc.md`. # Coming from MiniExcel MiniExcel moves rows of data and dictionaries in and out of XLSX and CSV with static methods and fills workbook templates. OneDocument has the same one-call shape for typed rows (`Spreadsheet.Read` / `Write`), a forward-only cursor for the lowest-level streaming, the full workbook model when you need cells and styles, and `Template.Render` for templates. Read [README.md](https://recursiveai.org/onedocument/docs/migration/) first for what is true on every page. ## Typed rows ```csharp // MiniExcel MiniExcel.SaveAs(path, orders); IEnumerable back = MiniExcel.Query(path); ``` ```csharp // OneDocument: Order is a partial class or record marked [SpreadsheetRow] Spreadsheet.Write(path, orders); // .xlsx, .csv or .tsv by extension IEnumerable back = Spreadsheet.Read(path); ``` MiniExcel binds by reflection at run time; OneDocument binds with code generated when you compile, so it works under trimming and NativeAOT. Column names, order, formats and ignoring are attributes on the members (`[SpreadsheetColumn]`, `docs/spreadsheets-poco.md`). Asynchronous forms: `Spreadsheet.ReadAsync` and `WriteAsync`. ## Untyped rows ```csharp // MiniExcel foreach (dynamic row in MiniExcel.Query(path)) { Console.WriteLine(row.A); } ``` ```csharp // OneDocument: one object array per row, no dynamic foreach (object?[] row in Spreadsheet.ReadRows(path, sheet: "Sheet1")) { Console.WriteLine(row[0]); } ``` ## Writing dictionaries and tables MiniExcel's `SaveAs` also takes dictionaries and `DataTable`/`IDataReader`. In OneDocument, write a typed record, or open a `Workbook` and set cells, or use the streaming writer: ```csharp // OneDocument: streaming write, constant memory using XlsxStreamWriter book = XlsxStreamWriter.Create(path); SheetStreamWriter sheet = book.AddSheet("Data"); foreach (Order o in orders) { sheet.BeginRow(); sheet.WriteNumber(o.Id); sheet.WriteText(o.Customer); } sheet.Complete(); book.Finish(); ``` ## Templates ```csharp // MiniExcel MiniExcel.SaveAsByTemplate(outPath, templatePath, new { Title = "Report", Items = items }); ``` ```csharp // OneDocument Template.Render(templatePath, new Report { Title = "Report", Items = items }, outPath); ``` The data type is marked `[TemplateModel]` (generated binding), or pass a dictionary, JSON or a `DataTable`. Tags are `{{ title }}` and `{% for item in items %}` with Excel formulas as expressions (`docs/templating.md`, `docs/templating-xlsx.md`); MiniExcel's own template syntax is not read, so templates are rewritten once. The same call fills `.docx` and `.pptx` templates. ## CSV ```csharp // MiniExcel MiniExcel.SaveAs("out.csv", orders); ``` ```csharp // OneDocument Spreadsheet.Write("out.csv", orders); // formula-injection safe by default; delimiter detected on read ``` CSV details: `docs/csv.md`. ## Not supported or different - No `QueryAsDataTable` or `GetSheetNames` statics: open a `Workbook` (`wb.Sheets`) or use `XlsxStreamReader`. - No dynamic rows and no `Dictionary` output from the typed binding. - No `MiniExcel.Insert` (append to an existing file); open the workbook, add rows, save. - No image or chart embedding option on `SaveAs`; the template engine inserts pictures (`{% image %}`). - Typed binding needs `partial` types marked `[SpreadsheetRow]`, so types you do not own cannot be bound directly; map them to a record you own. # Coming from ShapeCrawler ShapeCrawler edits PowerPoint files through a presentation object with slides and shapes. OneDocument's `Presentation` is the same idea on a lossless model, with templating and PDF output, and it reads legacy `.ppt`. The names below follow ShapeCrawler's published documentation; its API has changed between releases, so check the release you use. Read [README.md](https://recursiveai.org/onedocument/docs/migration/) first for what is true on every page. ## Open, read, save ```csharp // ShapeCrawler var pres = new Presentation(path); foreach (var slide in pres.Slides) { foreach (var shape in slide.Shapes) { if (shape.TextFrame is { } tf) { Console.WriteLine(tf.Text); } } } pres.Save(); ``` ```csharp // OneDocument using Presentation deck = Presentation.Open(path); // .pptx, .ppsx, .potx and legacy .ppt foreach (Slide slide in deck.Slides) { foreach (Shape shape in slide.Shapes) { if (shape.TextBody is { } body) { Console.WriteLine(body.Text); } } } deck.Save(); // atomic replace; only edited parts change ``` ## Edit text ```csharp // ShapeCrawler shape.TextFrame!.Text = "Quarterly results"; ``` ```csharp // OneDocument shape.TextBody!.SetText("Quarterly results"); // keeps the first run's formatting shape.TextBody.AddParagraph("Second line"); ``` Placeholder inheritance (master, layout, slide) is resolved for you: `paragraph.ResolveFormat()` returns the effective size, bullet and alignment (`docs/presentations.md`). ## Add, duplicate, move, delete slides ```csharp // ShapeCrawler pres.Slides.Add(layout); // add from a layout pres.Slides[0].Remove(); ``` ```csharp // OneDocument Slide added = deck.AddSlide("Title and Content"); // by layout name; or AddSlide(layout) added.SetPlaceholderText("title", "Results"); Slide copy = deck.DuplicateSlide(deck.Slides[0]); deck.MoveSlide(copy, 0); deck.DeleteSlide(deck.Slides[2]); ``` Relationships, content types, notes, sections and custom shows stay consistent. Speaker notes: `slide.SetNotesText("...")`. ## Tables ```csharp // OneDocument: an existing table shape foreach (Shape shape in slide.Shapes) { if (shape.Table is { } table) { /* rows, cells, merged cells */ } } ``` Tables are read and edited, and `slide.AddTable(rows, left, top, width)` creates one (positions and sizes are in points). ## Pictures ```csharp // OneDocument Shape logo = slide.AddPicture(File.ReadAllBytes("logo.png"), left: 560, top: 20, width: 72, altText: "Company logo"); ``` ## Templates and output Fill a deck from data, one slide per item, with the same engine as for documents and workbooks: ```csharp Template.Render(path, report, outPath); // outPath may be .pptx or .pdf ``` Slides render to PDF and PNG (`deck.SaveAs("deck.pdf")`, `Office.Convert`), with the Rendering package. ShapeCrawler does not render. ## Not supported or different - A slide gets pictures and tables through `AddPicture` and `AddTable`; there is no high-level call to add other shapes, charts or connectors. Existing ones are kept byte-identical and drawn; other new content comes from templates (`{% image %}`) or the typed views and markup. - No animation or transition API: they are preserved and never touched, not rendered. - No fluent shape-geometry setters (`shape.X = ...`): geometry is in the shape's typed view. - `.ppt` can be read and converted, not written. # Coming from Xceed DocX DocX loads a Word document with `DocX.Load`, exposes `Paragraphs`, `Tables`, `InsertParagraph`, `ReplaceText` and `AddTable`, and saves. OneDocument's `Document` covers the same ground on the lossless model, and adds templating, conversion and rendering. Read [README.md](https://recursiveai.org/onedocument/docs/migration/) first for what is true on every page. ## Load, read, add, save ```csharp // Xceed DocX using var doc = DocX.Load(path); foreach (Paragraph p in doc.Paragraphs) { Console.WriteLine(p.Text); } doc.InsertParagraph("Added"); doc.SaveAs(outPath); ``` ```csharp // OneDocument using Document doc = Document.Open(path); foreach (Paragraph p in doc.Paragraphs) { Console.WriteLine(p.Text); } doc.Body.AddParagraph("Added"); doc.SaveAs(outPath); ``` `Document.Create()` makes a new document. `Save()` replaces the opened file atomically. `Document.Open` also reads `.doc`, `.rtf`, `.odt` and Word 2003 XML, which DocX does not. ## Styles on paragraphs ```csharp // Xceed DocX paragraph.StyleName = "Heading1"; ``` ```csharp // OneDocument paragraph.Style = "Heading 1"; // by name; an unknown name throws, nothing is created doc.Body.AddParagraph("Title").Style = "Heading 1"; ``` Style definitions, inheritance and numbering are resolved by the hub (`docs/docx-styles-numbering.md`). ## Replace text ```csharp // Xceed DocX doc.ReplaceText("[name]", "Ada"); ``` ```csharp // OneDocument: find and replace across runs, keeping formatting doc.Replace("[name]", "Ada"); ``` The exact call shape, the search options (case, whole word, regular expression) and the report it returns are in `docs/operations-find-replace.md`. For filling placeholders use the template engine below, which is tolerant of tags that Word split across runs. ## Templates and mail merge ```csharp // Xceed DocX: placeholders replaced one by one doc.ReplaceText("{{customer}}", customer.Name); ``` ```csharp // OneDocument: tags, loops and conditions in the document, data from an object, JSON, a dictionary or a DataTable Template.Render(path, invoice, outPath); // outPath may be .docx or .pdf ``` Syntax: `{{ customer.name }}`, `{% for line in lines %}` ... `{% endfor %}`, `{% if overdue %}`; expressions are Excel formulas (`docs/templating.md`, `docs/templating-docx.md`). Word's own merge fields are filled by `doc.MailMerge(data)` (`docs/mail-merge.md`). ## Tables ```csharp // Xceed DocX Table t = doc.AddTable(2, 3); t.Rows[0].Cells[0].Paragraphs[0].Append("A1"); doc.InsertTable(t); ``` ```csharp // OneDocument Table t = doc.Body.AddTable(2, 3); // or AddTable(rows) from strings t.SetCellText(0, 0, "A1"); t.InsertRow(1); ``` `AddTable` adds the table at the end of the body in the Table Grid style, at the page's text width. Existing tables are fully editable (rows, columns, merged cells through the logical cell model). ## Images, headers, footers, properties Pictures: `paragraph.AddPicture(bytes, widthPoints, heightPoints, altText)` (or `body.AddPicture`) adds an inline picture; the template engine inserts pictures too (`{% image %}`), and existing pictures are kept and rendered. Links: `paragraph.AddHyperlink("text", "https://...")`. Headers and footers: `doc.Sections[0]` exposes them (`docs/docx-tables-sections.md`). Core properties (`Title`, `Creator`, ...) are on the document's properties object (`docs/docx-stories-properties.md`). ## Output formats `doc.SaveAs("x.pdf")`, `"x.rtf"`, `"x.odt"`, `"x.html"`, `"x.md"` pick the writer by extension. Conversions that lose something report it (`doc.ImportLoss` for what was lost reading, `doc.ExportLoss` for writing RTF; `docs/rtf.md`, `docs/html.md`). ## Not supported or different - No fluent chaining of formatting (`Bold().Color(...)` on appended text). Formatting is set through properties of the run returned by `paragraph.AddRun("text")`: `run.Bold = true; run.Color = "C00000";`. - No `AddHeaders` helper on the hub; headers and footers are reached through `doc.Sections`. - No `DocX.Create` + `InsertParagraph` of formatted text in one call; the paragraph is added, then its runs are added and formatted. - Revisions are a full model (`doc.Revisions`: read, author, accept, reject), which DocX does not have. # Authoring with the hub The hub types add content through small calls that write through the markup tree and the generated views. Everything else in the part stays as it was: unknown elements, attributes, prefixes, `mc:AlternateContent` and `extLst` keep their place, and the parts an authoring call does not touch are written byte-identical. `samples/Authoring` builds a document, a workbook and a deck with these calls; `testdata/authoring` holds the three files it writes. Errors for bad arguments are `ArgumentException` (a value that cannot be stored) or `ArgumentOutOfRangeException` (a count or index); a name that does not exist is a typed `OfficeException` (below). A call that throws leaves the document as it was. ## Documents **Styles by name.** `Paragraph.Style`, `Run.Style` and `Table.Style` read and set the style by the name Word shows. Built-in names are stored in lower case in files (`heading 1`) and read back capitalised (`Heading 1`); matching ignores case. A paragraph without a style reads as the default paragraph style (`Normal`). Setting a name the document does not define throws `StyleNotFoundException` (`docx.style-not-found`) and creates nothing; a name that belongs to another type of style is the same error. `StyleId` reads and writes the id as before. A document made by `Document.Create()` has only `Normal`. `doc.Styles.AddBuiltIn("Heading 1")` adds a built-in style under its well-known name when the document lacks it (Title, Heading 1 to 3, Caption, List Paragraph, Hyperlink, Table Grid) and returns the existing one otherwise. These definitions are plain: fonts are not tied to the theme. **Runs and formatting.** `paragraph.AddRun(text)` appends a run (tabs and line feeds become `w:tab` and `w:br`) and returns it. `Run.Bold`, `Italic` and `Underline` are `bool?`: true turns the property on, false writes an explicit off (so it overrides the style), null removes the direct setting. `FontSize` is in points (written in half-points, also for complex scripts), `Color` is six hex digits or `auto`, `FontName` names the font for Latin, East Asian and complex-script text. The properties report direct formatting only; resolved formatting (the style's included) comes from the style sheet (`docs/docx-styles-numbering.md`). **Links and pictures.** `paragraph.AddHyperlink(text, address)` adds a hyperlink in the `Hyperlink` character style (added when missing). `#name` is a bookmark in the document; anything else becomes an external relationship of the main part. It is stored and never fetched. `paragraph.AddPicture(bytes, widthPoints, heightPoints, altText)` and `body.AddPicture(...)` add an inline picture (PNG, JPEG, GIF or BMP, recognised by header; the pixels are not decoded). With one side given the other keeps the picture's proportions; with none the picture shows at its pixel size on a 96 dpi screen, scaled down to fit 6 inches. Drawing ids (`wp:docPr/@id`) continue after the largest in the main part. **Tables.** `body.AddTable(rows)` takes rows of strings (a `string[][]`, lists, `string?[,]`), `body.AddTable(rowCount, columnCount)` an empty one. The table goes at the end of the body, before the last section's properties, in the Table Grid style with columns of equal width spanning the page's text width (page width less the margins of the last section; 6.5 inches when there are none). Short rows get empty cells. At most 63 columns, as in Word. `table[row, column]` reaches a cell (a merged region answers to any of its grid positions), `table.SetCellText` sets its text. ## Workbooks **Ranges.** `sheet.Range("A1:C10")` (or `Range(CellRange)`) returns a `SheetRange`, a handle over the sheet. `Values` reads and writes a two-dimensional array, rows first. Reading gives `null` (empty), `double`, `DateTime` (numbers whose format shows a date), `string`, `bool` or `CellError`. Writing accepts those plus the integer and floating types, `decimal`, `DateOnly` and `CellValue`; `null` empties the cell and keeps its style. Strings are text and are never parsed as numbers or formulas (set `Cell.Formula` for those). The array must have exactly the range's size, every element is checked first, and a failing write changes nothing. `SetStyle`, `UpdateStyle` (one change function call per distinct current style) and `NumberFormat` apply to every cell; `Clear()` removes the cells completely and unmerges merged ranges inside; `Merge()` keeps the top-left value and empties the others, as Excel's Merge Cells does, and refuses a single cell or an overlap with an existing merge (`InvalidOperationException`). Reads and writes over more than 4,194,304 cells throw `InvalidOperationException`, so `A1:XFD1048576` cannot allocate by accident. **Pictures.** `sheet.AddPicture(bytes, "E1", widthPixels, heightPixels, altText)` stores the picture as a media part, creates the sheet's drawing when it has none and anchors the picture at the cell with the size given (pixels at 96 dpi; the other side keeps proportions). It returns the shape id. ## Presentations `deck.AddSlide("Title and Content")` adds a slide from the layout of that name (case ignored, every master searched); an unknown name throws `PresentationAuthoringException` (`pptx.layout-not-found`) listing the names there are. `slide.SetPlaceholderText("title", text)` sets a placeholder by type (`title` also finds `ctrTitle`; `subTitle`, `body`, `obj`, ...) and `SetPlaceholderText(1, text)` by index; lines become paragraphs. No such placeholder throws `PresentationAuthoringException` (`pptx.placeholder-not-found`) listing the placeholders the slide has. `slide.AddPicture(bytes, left, top, width, height, altText)` and `slide.AddTable(rows, left, top, width, rowHeight)` (or `AddTable(rowCount, columnCount, ...)`) put a picture or a table on top of the slide. Positions and sizes are in points; the table's columns share the width equally and it uses PowerPoint's default table style with a header row. Shape ids continue after the largest on the slide. New shapes go at the end of the shape tree, before a trailing `extLst`. ## Sanitising in memory `doc.Sanitize()`, `wb.Sanitize()` and `deck.Sanitize()` run the sanitiser of `Office.Sanitize` on the open file and return its `SanitizeReport`. The default categories apply unless a `SanitizeOptions` is passed. Pending edits are committed first. The sanitiser works on a copy, **verifies on the call** (the result is inspected again, and `SanitizeVerificationException` is thrown if a category reported as removed still holds something), and only then is the file's package brought to the result: parts the sanitiser did not change keep their stored bytes, so a later save writes them raw. Verification runs on the call and not on save so that a failure surfaces where the caller can handle it, and the document is left as it was. After the call, objects obtained from the file before it (paragraphs, sheets, slides, the style sheet) no longer belong to it; read them again. The file's `Format` does not change: when the report's `OutputFormat` differs (removing a macro project makes a macro-enabled file macro-free), name the output file after it. A binary workbook (XLSB) cannot be sanitised in memory (`OfficeUnsupportedException`); `Office.Sanitize(input, output)` writes a new file for it. For a copy of a file on disk, `Office.Sanitize(input, output)` stays the way: it never changes the input. # WordprocessingML styles and numbering How the document hub resolves formatting and list labels. References are to ECMA-376 Part 1 (5th edition) unless noted. Where Word's behaviour is not settled by the text, the rule below is what the hub does today and the item is marked **to check against Word**. ## Resolved properties (§17.7.2) Layers, lowest first. Each property keeps the layer (and style id) that set it. | Paragraph | Run | |---|---| | document defaults (`w:pPrDefault`) | document defaults (`w:rPrDefault`) | | table style: whole table, then regions | table style run properties | | numbering level `w:pPr` (when the *style* supplies the list) | | | paragraph style chain, base first (default paragraph style when none is named) | paragraph style chain `w:rPr` | | numbering level `w:pPr` (when the paragraph names the list itself) | character style chain (default character style when none is named) | | direct `w:pPr` | direct `w:rPr` | - A property is an element. `spacing`, `ind`, `rFonts` and `lang` merge attribute by attribute; a directly written `ascii` replaces an inherited `asciiTheme` (likewise `hAnsi`, `eastAsia`, `cs`), `firstLine` replaces an inherited `hanging` and the reverse, `left`/`start` and `right`/`end` are one edge. Everything else is replaced whole. - Borders (`pBdr`) merge per side, tab stops (`tabs`) per position (a `clear` removes the stop), the list reference (`numPr`) per child. - **Toggle properties (§17.7.3)** `b bCs i iCs caps smallCaps strike dstrike outline shadow emboss imprint vanish`: direct formatting sets the value outright; otherwise the values the table style, paragraph style and character style levels set are XORed (inside one level a `basedOn` chain overrides normally); with none, the document default stands. **To check against Word:** the XOR runs over the three style levels only, and the document default is not part of it. - A document without `w:pPrDefault` reads as 8 pt (160) space after when nothing else sets it (`ResolvedParagraphProperties.SpaceAfter`); observed in Word 365 (16.0.20527). Word's answers for the precedence case (document defaults, `basedOn` chains, the toggle XOR, direct formatting over styles, character style over paragraph style) and for the numbering label cases all match the resolver. Line spacing without `w:pPrDefault` is not settled: Word reported a multiple of about 1.16 lines. - A style chain follows `w:basedOn`. A missing base, a cycle and an unknown style id are defects (`docx.style-*`) and never exceptions. - Table style regions apply in the order wholeTable, band1Vert, band2Vert, band1Horz, band2Horz, firstCol, lastCol, firstRow, lastRow, nwCell, neCell, swCell, seCell (§17.7.6). `w:tblLook` comes from its attributes or its hex mask; absent means no region flags. The header row is not banded and banding counts from the first row after it. `w:cnfStyle` caches are ignored. - Theme fonts and colours resolve from the theme part with the default colour map. A colour's tint and shade are not applied (the raw attributes stay reachable) — **needs Word output**. - **Numbering level versus paragraph style (to check against Word):** when the style supplies `numId` the level's `w:pPr` sits below the paragraph style; when the paragraph names the list directly it sits above it. ## Numbering (§17.9) - A paragraph's list is its direct `numId`, else the nearest style in its chain with one; `numId` 0 turns numbering off. The level is the direct `ilvl`, else the style's, else the level whose `w:pStyle` names the paragraph's style, else 0. - Counters belong to the abstract definition (after `w:numStyleLink`), so instances of one definition continue each other. An instance with a `w:startOverride` restarts that level the first time the instance is used. A `w:lvlOverride/w:lvl` replaces the level for that instance. LibreOffice's text export agrees on every generated case. - The first number of a level is the override, else `w:start`, else 0. - After a paragraph at level *L*, level *j > L* starts again when *L < lvlRestart* (one-based `w:lvlRestart`; default *j* zero-based, i.e. after any higher level). `lvlRestart` 0 never restarts. **To check against Word:** LibreOffice ignores `lvlRestart` 0 (it restarts); the hub follows the specification. - Label text fills `%1`..`%9` with the counter of that level in that level's format; legal numbering (`w:isLgl`) shows the other levels in decimal. A level that was never used shows its start value. - Formats produced exactly: `decimal decimalZero upperRoman lowerRoman upperLetter lowerLetter ordinal bullet none decimalEnclosedCircle (1-20) decimalFullWidth decimalHalfWidth`. Letters repeat (`z`, `aa`, `bb`). Text forms (`cardinalText`, `ordinalText`), `chicago`, `hex` and every script-specific format are produced as a best guess and every label that uses one says `NeedsWordOutput`. - Defects: `docx.num-missing`, `docx.num-abstract-missing`, `docx.num-level-missing`, `docx.num-link-broken`. ## Editing Style and numbering edits change only `styles.xml`, `numbering.xml` and `document.xml`: a part is rewritten only when its tree was edited, and missing parts are created with their relationship. New elements use the part's own prefix and respect the schema's sequence (abstract definitions before numbering instances). # WordprocessingML tables, sections, headers and footers How the document hub reads and edits them. References are to ECMA-376 Part 1 (5th edition) unless noted. Items marked **to check against Word** are what the hub does today where the text leaves the answer open. ## Tables (§17.4) - **Rows and cells** are enumerated through content-control and custom-XML wrappers, so nothing is hidden. A cell is a block container (`TableCell`); `w:tcPr` is not a block. - **Logical grid** (`Table.Logical()`): rows × grid columns. A cell starts at the running column (after `w:gridBefore`) and covers `w:gridSpan` columns (below 1 or above 16384 reads as 1). The grid width is the larger of `w:tblGrid` and the widest row, so rows wider than the grid widen it (`docx.table-grid-mismatch`). - **Vertical merge** (§17.4.85): `w:vMerge` with `w:val="restart"` starts a region; with `continue` or no value it continues the cell directly above in the same starting grid column. A continuation under a cell that is not part of a merge still merges upward: the cell above becomes the region's start (observed in Word 365, 16.0.20527: three rows of `vMerge` continuations under an unmerged first cell read as one cell). The file is malformed, so the repair is reported as `docx.table-merge-orphan`. A continuation in the first row, or with no cell starting in its column above, is a plain cell. - **Horizontal merge**: current files use `w:gridSpan`. The legacy `w:hMerge` (restart/continue) widens the cell to its left; an orphan continuation is a plain cell. - A region is one `LogicalCell` anchored at its top-left cell; the swallowed physical cells are in `Covered`. Empty grid positions (`gridBefore`, `gridAfter`, short rows) read as null. - **Table styles**: `Table.Look` reads `w:tblLook` (attributes or the hex mask). `StyleSheet.GetTableRegions` names the conditional regions of a logical cell in application order (§17.7.6); the header row is not banded and banding counts from the row after it. A merged cell counts by its top-left position for banding. `firstCol` and `lastCol` follow the row's first and last *physical* cell, so a row that starts after `gridBefore` still has its first cell in the first column (observed in Word 365, 16.0.20527). Band regions still count by grid position; Word's behaviour for bands in rows with merged or gapped cells has not been verified. `ResolveTableCell` layers style table and cell properties, regions, the table's own properties and the cell's own; borders and margins are flattened per side (`tblBorders/top`, `tcBorders/left`, `tblCellMar/start`). `w:tblLook` attributes win over the hex mask when both are present (observed in Word 365, 16.0.20527). ### Edits Edits change only the table's markup; properties, unknown markup and untouched rows stay byte-identical, and only the part that holds the table is rewritten. - `InsertRow(index)` copies the row above (the first row at the top), clears the cells to one empty paragraph that keeps the first paragraph's properties, removes `w:ins`/`w:del`/`w:trPrChange` from the copy and the paragraph ids. A new cell continues a vertical merge exactly when the cells above and below belong to one region. - `DeleteRow(index)` makes the next row's cell the start of a region that the removed row started. The only row cannot be deleted. - `InsertColumn(column)` widens cells that span the insertion point, adds a plain empty cell elsewhere (copying the neighbour's width) and a `w:gridCol` of the left neighbour's width. Inside `gridBefore`/`gridAfter` the counts change instead. - `DeleteColumn(column)` narrows spanning cells, removes cells that cover only that column, and removes the `w:gridCol`. The only column cannot be deleted. - `SetCellText` keeps the first paragraph's properties and its first run's formatting and removes every other block of the cell. ## Sections (§17.6) - A section ends at the paragraph whose `w:pPr` holds a `w:sectPr` (top level or in a block content control); the last section's `w:sectPr` closes the body. A body without one has one section of defaults. - Defaults when attributes are absent: Letter portrait (12240 × 15840), margins 1440, header and footer distance 720, gutter 0, one column with 720 space, `w:type` next page. An unusable `w:pgSz` reads as Letter (`docx.section-page-size-invalid`). - Columns: for unequal layouts the `w:col` list defines the count (`docx.section-columns-mismatch` when `w:num` differs). - Page numbering, line numbering, document grid, `w:titlePg`, `w:bidi`, `w:vAlign`, `w:textDirection`, `w:noEndnote` are read as data. Setters exist for page size, margins, break type, title page, page numbering and equal columns; they respect the schema sequence and create the body's `w:sectPr` when missing. ## Headers and footers (§17.10.5) - A reference is `w:headerReference`/`w:footerReference` with `w:type` `default`, `first` or `even` and an `r:id` of the main part. A reference that does not lead to a part is ignored (`docx.header-missing`); a second of the same kind and type is ignored (`docx.header-reference-duplicate`). - **Linked to previous**: a section without its own reference of a kind and type uses the nearest earlier section's. The first section without one prints blank. There is no fallback between types: a missing `first` or `even` story is blank (or inherited for that type), never the default story. - **Page choice** (`Section.ForPage`): with `w:titlePg`, the first page of the section uses `first`; else with `w:evenAndOddHeaders` (settings part), an even *printed* page number (after any `pgNumType` restart, not the physical page) uses `even`; else `default`. `Section.FirstPrintedPageNumber` gives the number a section's first page prints. - **Blank pages** (`Section.RequiresBlankPageBefore`): observed in Word 365 (16.0.20527), with `w:evenAndOddHeaders` on a new-page section whose first printed number has the other parity from the physical page it would start on starts one page later, after an empty page (restart at 1 after three pages: starts on physical page 5; restart at 2 after two pages: physical page 4). Layout inserts that page; the hub only answers the question. - **To check against Word:** a section with `w:titlePg` and no `first` reference of its own, after a section that has one, inherits it (the hub), while LibreOffice prints the default header on that page. - `Section.AddOwn` creates `/word/headerN.xml` (or footer) with one empty paragraph, the relationship and the reference. `Document.EvenAndOddHeaders` writes `w:evenAndOddHeaders` into settings, creating the part when needed. # WordprocessingML fields, notes and comments How the document hub reads and edits fields, footnotes and endnotes, and comments. The model holds codes and cached results; evaluating a field that needs layout (page numbers, tables of contents) is not done here. ## Fields `Document.Fields()` walks the body once and lists every field in the order its code begins, so an outer field comes before the fields nested in it. `Document.Fields(container)` does the same for a note, a comment or any other block container. - **Simple** (`w:fldSimple`): the instruction is the `w:instr` attribute; the result is the content. `w:dirty` and `w:fldLock` are exposed. - **Complex** (`w:fldChar` begin / `w:instrText` / separate / result / end): one field however many runs, paragraphs or cells it crosses, and however deep it nests. The instruction is the field's own `w:instrText`, concatenated, plus the cached result of fields nested inside the code. The result text is the text of its runs including nested results; tabs read as `\t` and breaks as `\n`. `w:delText` and `w:delInstrText` count for nothing. - The `mc:Fallback` branch of alternate content is skipped, because it repeats the chosen branch. - **Instruction syntax** (`FieldInstruction`): the first word is the type (upper-cased); quoted strings may use straight or curly quotes, and a backslash inside quotes escapes the next character; `\x` starts a switch, with any glued text as its argument (`\*MERGEFORMAT`). A switch takes the next word as its argument unless it is a known flag of that field type (the table is in the source, from the field definitions). `=` fields keep the expression as one argument. Parsing never fails; an unclosed quote sets `IsWellFormed` to false. - **Editing**: `SetInstruction` writes the text exactly as given. `SetResult` keeps the first plain result run (and its formatting), puts the text in it and removes every other result run, nested fields included; a field without a separator gets one and a run formatted like the begin character. A result that crossed paragraphs leaves those paragraphs in place without the removed runs. - Defects: `docx.field-unbalanced`, `docx.field-too-deep`. ## Notes `Document.Footnotes` and `Document.Endnotes` list every note of the part, separators included (`Note.Type`). A document without the part has an empty collection. - `Add(text)` creates the part when missing (with the separator and continuation-separator notes, ids -1 and 0), the relationship and content type, and the settings' list of separator ids when a settings part exists. The first normal note has id 1. The paragraph starts with the automatic number mark, styled when the document defines the note styles. - `Note.InsertReference(run)` / `AppendReference(paragraph)` insert a reference run (character style when defined, else superscript). `Remove(note)` removes the note and all its references; separators cannot be removed or referenced. - `NoteNumbering` (per section `w:sectPr` or document-wide in the settings) reads and writes `pos`, `numFmt`, `numStart`, `numRestart` in schema order; `Combine` applies section over document over defaults. With no `numFmt` anywhere, footnotes are `decimal` and endnotes `lowerRoman` (observed in Word 365, 16.0.20527; the schema's default is decimal for both). - Defects: `docx.note-missing`, `docx.note-duplicate-id`. ## Comments `Document.Comments` reads `comments.xml` with the modern parts that go with it, when present: `commentsExtended` (parent link by the `w14:paraId` of the last paragraph, and the done state), `commentsIds` (durable ids), `commentsExtensible` (UTC date) and `people`. - `Add(first, last, author, text, ...)` writes the comment, the range start before `first`, the range end after `last`, and the reference run after the end. The date is only written when the caller passes one: the library never reads the clock. Paragraph ids and durable ids are deterministic (derived from the comment id, moved on when taken). - `Reply(parent, ...)` anchors the reply beside the parent's markers (start after the parent's start, end and reference after the parent's reference) and writes the parent link, creating `commentsExtended` when needed. A reply to a reply goes to the top-level comment. - `Remove(comment)` removes its replies, markers, reference runs, and its entries in every modern part; the `people` entry stays. - Add and remove change the comment parts, `document.xml` and the relationships/content types of a new part, nothing else. - Defects: `docx.comment-duplicate-id`, `docx.comment-missing`, `docx.comment-range-unmatched`. ## Questions for desktop Word 1. Endnote default number format with no `w:numFmt` (see above). 2. Whether a comments part without `mc:Ignorable="w14"` that gets `w14:paraId` added to an existing part (declared locally on the element) opens without repair; new parts declare it on the root. 3. Whether `commentsExtended` alone (no `commentsIds`) is accepted for a reply thread. ## New note and comment ids `Footnotes.Add`, `Endnotes.Add` and `Comments.Add` take one more than the highest id in use, and at least 1 for notes and 0 for comments. "In use" covers the notes or comments part (separator and continuation notes included, ids in any `mc:AlternateContent` branch included), the main part (references and comment ranges, also to notes that do not exist, since a dangling reference would attach to the new note) and the settings part (which names the separator notes). Notes and comments added earlier in the same session are in the part and so count. Ids of removed notes may be reused only when nothing refers to them. # Content controls (structured document tags) `Document.ContentControls()` enumerates every `w:sdt` of the main part and of its headers, footers, footnotes and endnotes in document order (a control before the controls nested in it). `BlockContentControl.Control` and `InlineContentControl.Control` reach the same model from the block and inline hub objects. A `ContentControl` is a thin wrapper over the element: reads go to the markup tree, edits change only the control's own markup, and everything unknown (extension elements, `mc:` markup, attributes, prefixes, order) stays. ## Read | Member | Source | |---|---| | `Level` | Block, Inline, Row (`w:sdt` in `w:tbl`), Cell (`w:sdt` in `w:tr`). A control nested in another takes its place from the wrappers above it. | | `Type` | The one type element of `w:sdtPr`: `richText`, `text`, `picture`, `date`, `dropDownList`, `comboBox`, `group`, `citation`, `bibliography`, `equation`, `docPartObj`, `docPartList`; `w14:checkbox`; `w15:repeatingSection`, `w15:repeatingSectionItem` (these two win when they sit beside another type). No type element reads as rich text, and `IsTypeExplicit` says so. | | `Tag`, `Alias`, `Id`, `Lock`, `IsTemporary`, `IsShowingPlaceholder`, `PlaceholderDocPart` | `w:tag`, `w:alias`, `w:id`, `w:lock`, `w:temporary`, `w:showingPlcHdr`, `w:placeholder/w:docPart`. | | `ListItems`, `SelectedIndex`, `LastValue` | `w:listItem` (display text defaults to value and the reverse); the selection is the item whose display text equals the content, else the one whose value is `w:lastValue`. | | `IsChecked` | `w14:checked` (a check box without it is unchecked). | | `Date`, `DateFormat` | `w:date/@w:fullDate`, `w:dateFormat`. | | `Text` | `w:t`, `w:tab` (`\t`), `w:br`/`w:cr` (`\n`), and `\n` between paragraphs. Deleted text, drawings, field instructions and `mc:Fallback` are not text. | | `Children()`, `Parent` | Controls nested directly in, or around, this one. A repeating section's children are its items. | | `Binding`, `EvaluateBinding()` | `w:dataBinding`; see below. | ## Edit - `SetText(text)` replaces the content. Inline: one run with the first run's properties, `w:tab` for `\t`, `w:br` for `\n` (rich text and multi-line plain text; a single-line plain text control turns `\n` into a space). Block: one paragraph per line, each with the first paragraph's properties. A showing placeholder is cleared (`w:showingPlcHdr` removed, the placeholder run style not copied). A drop-down list only takes one of its items, by display text or value. A character XML cannot carry throws `ArgumentException` before anything is changed. - `SelectItem(index)` sets the content to the item's display text and `w:lastValue` to its value. - `SetChecked(bool)` sets `w14:checked` and the glyph (`w14:checkedState`/`w14:uncheckedState`, ☒ and ☐ by default) in the first `w:t`, so the run's symbol font stays. - `RemoveKeepingContent()` replaces the control with the children of its `w:sdtContent` (blocks, runs, rows or cells). `RemoveWithContent()` removes both. - Row-level and cell-level controls, check boxes, pictures, groups and repeating sections have no text; `SetText` throws `InvalidOperationException` for them. - Editing does not push data into the bound custom XML part, and a bound control's content is not refreshed from it. Applying bindings is the template engine's job. ## Data bindings and the XPath subset `CustomXmlParts` lists the parts the main part relates to with `customXml`, with their store item ids (`ds:itemID` of the related `customXmlProps` part). A binding names its store by `w:storeItemID`, matched case-insensitively. The id `{6C3C8BC8-F283-45AE-878A-BAB7291924A1}` is the package's core properties (`/docProps/core.xml`, found by the package relationship); Word-written files use it for title, subject and similar controls. `EvaluateBinding()` reads the current value of the bound node. It never changes anything, and no script runs. The path is evaluated by an XPath 1.0 subset (`XPathSubset`): - absolute and relative location paths of child steps (`/ns0:root[1]/ns0:name[1]`), `//` descendant steps, `.`, `..`, `*`, `@attr` and `text()`; - predicates: 1-based position `[n]`, `last()`, comparisons (`=`, `!=`, `<`, `>`, `<=`, `>=`) between paths, strings and numbers, `and`, `or`, `not()`, `name()`, `local-name()`, `contains()`, `starts-with()`; - prefixes come from `w:prefixMappings` (`xmlns:ns0='uri' …`); an unprefixed name matches no-namespace elements, as in XPath 1.0. The value is the element's text content, the attribute's value, or the element's own text nodes for `text()`. Results (`BindingStatus`): `Resolved`, `NoMatch`, `StoreNotFound`, `UnsupportedPath` (other axes, functions and operators outside the list, and anything that is not a node set at the top), `Invalid` (no path, malformed, a prefix the mappings do not declare). ## Binding data `Document.BindContentControls(data)` writes data into the custom XML parts and shows it in the controls; `RefreshContentControls()` re-reads the parts into the controls; `PushContentControls()` is the reverse (controls into parts). `data` is anything the template engine reads, or XML given as a string or bytes. Each control's `w:dataBinding` path is read as a path into the data: the root element's name is not matched, names match ignoring case, `[n]` takes the n-th item of a list. A list in the data may stand for the container of the repeated element (JSON `lines: [...]` for `lines/line`). The value is written into the bound node (created when missing) and the control shows it. Controls with no value in the data keep their content, and nothing outside the custom XML parts and the controls changes. | Control | Shown value | Written back | |---|---|---| | plain text, rich text | the text (a rich control keeps its first run's formatting) | the text | | date | the value formatted with `w:dateFormat` | the full date, `yyyy-MM-ddTHH:mm:ssZ` | | check box (w14) | the check state; `1`, `true` and `on` are checked | `1`/`0` (`true`/`false` when the node used those) | | drop-down, combo box | the list item whose value (or text) matches; a combo keeps a value not in its list | the item's value | | attribute binding (`@attr`) | the attribute | the attribute | A repeating section (`w15:repeatingSection`) bound to a repeating node takes the first item as the pattern and is copied or trimmed to the number of nodes in the list, with the paths inside re-indexed (`[1]` becomes `[n]`); nested repeating sections work. A placeholder-showing control writes nothing on push. ## Limits and defects Enumeration, text reading and `Children()` walk the tree iteratively, so nesting depth is bounded only by the XML depth limit. Reading repairs nothing. Binding reports `binding.` defects in `ContentBindingReport.Warnings` (`binding.store-not-found`, `binding.unsupported-path`, `binding.no-data`, `binding.unsupported-control`, `binding.value-not-in-list`, `binding.no-template-item`; see `defects.md`). Unreadable custom XML parts are reported by the part cache as `docx.part-unreadable` and the bindings to them read as `StoreNotFound`. # Text boxes, stories, list labels and document properties ## Text boxes `Document.TextBoxes()` lists the text boxes of every story (body, headers and footers, footnotes, endnotes, comments); `BlockContainer.TextBoxes()` and `Paragraph.TextBoxes()` list the ones anchored in one container or paragraph. A `TextBox` is a `BlockContainer` over one `w:txbxContent`, so its paragraphs and tables, edits and tracked changes work like those of the body. Paragraphs of a box are never among the paragraphs of the container that holds the shape. A box inside a box is reached through the outer box. Boxes found before an operation that replaces elements (accepting or rejecting changes) should be listed again. `Document.WrapParagraph(MarkupRef)` gives the hub paragraph for any `w:p` of the document. - **Kinds:** DrawingML (`wps:txbx`) and VML (`v:textbox`). `Anchor` is the paragraph that holds the shape, `Shape` the shape element, `HostStory` the story it sits in. - **`mc:AlternateContent`:** Word writes a DrawingML box twice, in `mc:Choice` and in a VML `mc:Fallback`. The box exposes the branch a modern reader takes (the first `mc:Choice` whose required namespaces are understood, else the `mc:Fallback`, ECMA-376 Part 3 §10.2.4), listed once. `Alternate` is the matching content in the other branch (pairing is by order of boxes inside the block). When the exposed content changed since the box was found, `CommitChanges` (or `SyncAlternate()`) replaces the alternate with a copy, so both branches stay equal. A box nobody edited leaves its alternate untouched. - **Tracked changes:** `Revisions.CreateAuthor(...)` accepts the paragraph nodes of a box. After the copy, the revision list shows each change twice (once per branch, same id); accept and reject act on both. Word reads the exposed branch only and counts each change once. - **Chains:** `TextBox.Chain` lists the shapes in reading order. DrawingML: the first shape holds `wps:txbx id=N`, followers hold `wps:linkedTxbx id=N seq=k` and no content (`TextBoxFrame.Story` is null). VML: `mso-next-textbox:#` in the style of `v:textbox`; every shape has its own content. ## List labels `NumberingDefinitions.LabelStories()` counts list numbers the way Word does, observed on generated documents in Word 16.0.20527 (`tools/office-oracle/make-story-cases.py`, probe key `stories`): | Flow | Rule | |---|---| | Body | its own count | | Headers and footers | one flow for all sections: section by section; inside a section even header, default header, even footer, default footer, first header, first footer | | Footnotes | one flow across all notes | | Endnotes | one flow across all notes | | Comments | one flow across all comments | | Text boxes of the body | one flow in document order; the VML copy is not counted | Counters never carry from one flow into another. Text boxes anchored in a header, note or comment form one flow per kind of host; this one is not checked against Word. `DocumentText.ListLabels` covers all of these, so extracted text carries the labels of every story. ## Document properties `Document.Properties` (core namespace `RecursiveAI.OneDocument.Properties`, usable on any package) reads and writes `docProps/core.xml`, `app.xml` and `custom.xml`. Parts are created with their relationship and content type on the first write and are left byte-identical when not edited. Edits keep other elements in order. The library never reads the clock: dates are written as given, UTC to the second (`xsi:type="dcterms:W3CDTF"` on the dcterms dates), and no date changes by itself. Custom values: text (`vt:lpwstr`), `vt:i4`, `vt:i8`, `vt:r8`, `vt:bool`, `vt:filetime`. Other types are read as text and kept. Names compare ignoring case. A new property gets the next `pid` after the highest (the format identifier is fixed by [MS-OSHARED]); ids of other properties never change. ## Content types A part's content type is its `Override`, else the `Default` for its extension (ECMA-376 Part 2 §10.1.2). A package whose main part has no `Override` is recognised by the format sniffer and opens. # Revisions: tracked changes in WordprocessingML This note states what the revision model (`RecursiveAI.OneDocument.Documents.Revisions`) does. The XML documentation on each type states the details. Section numbers are ECMA-376 Part 1 (5th edition) unless noted. ## What is modelled A revision is an element in the main namespace that records a change by an author at a date (`w:id`, `w:author`, `w:date`; the date is optional) (§17.13.5). The model reads them from any set of parts: the main document, headers, footers, footnotes, endnotes, comments, styles and numbering. | Kind | Element | Where | |---|---|---| | Insertion, Deletion | `w:ins`, `w:del` | around runs and other inline content (§17.13.5.18, §17.13.5.14) | | MoveFrom, MoveTo | `w:moveFrom`, `w:moveTo`, with `w:moveFromRangeStart/End`, `w:moveToRangeStart/End` | around runs; the range markers carry the name that pairs source and destination (§17.13.5.22, §17.13.5.25) | | ParagraphMarkInserted / Deleted / MovedFrom / MovedTo | `w:ins`, `w:del`, `w:moveFrom`, `w:moveTo` | in the run properties of the paragraph mark (`w:pPr/w:rPr`) | | RunProperties, ParagraphProperties, SectionProperties, TableProperties, TablePropertyExceptions, TableGrid, TableRowProperties, TableCellProperties changes | `w:rPrChange`, `w:pPrChange`, `w:sectPrChange`, `w:tblPrChange`, `w:tblPrExChange`, `w:tblGridChange`, `w:trPrChange`, `w:tcPrChange` | last child of the property container; holds the previous properties | | TableRowInserted / Deleted | `w:ins`, `w:del` | in `w:trPr` | | CellInserted / Deleted / Merged | `w:cellIns`, `w:cellDel`, `w:cellMerge` | in `w:tcPr` | | NumberingChange, NumberingInserted | `w:numberingChange`, `w:ins` | in `w:numPr` | Moved-away text stays `w:t` inside `w:moveFrom`; only `w:del` uses `w:delText`. A `w:moveFrom` holding `w:delText` makes Word 365 (16.0.20527) offer to repair the file, so authoring writes `w:t` there; reading accepts either. The same element name means different things by its parent, so a revision's kind comes from the element and its parent together. Strict and Transitional namespaces are read alike. Each revision has an author, a date (UTC; a date without a zone is taken as UTC), an id, a scope (content, paragraph mark, formatting, table, numbering), the part, the element and, for content, its text. A revision with a missing or unusable `w:id`, `w:author` or `w:date` is still listed and reported as a defect (`docx.revision-id-invalid`, `docx.revision-author-missing`, `docx.revision-date-invalid`). ## Accepting and rejecting | Revision | Accept | Reject | |---|---|---| | Insertion, MoveTo | the content stays, the wrapper goes | the wrapper and content go | | Deletion, MoveFrom | the wrapper and content go | the content stays; `w:delText` becomes `w:t` and `w:delInstrText` becomes `w:instrText` | | Paragraph mark inserted or moved-to | the marker goes | the mark goes: the paragraph's content joins the next paragraph (see below) | | Paragraph mark deleted or moved-from | the mark goes: the content joins the next paragraph | the marker goes | | Property change | the record goes; the current properties stay | the properties are replaced by the recorded ones; entries that are not part of the change stay (see below) | | Row inserted / deleted | marker goes / the row goes | the row goes / marker goes | | Cell inserted / deleted | marker goes / the cell goes | the cell goes / marker goes | | Cell merge | the vertical merge becomes `vMerge` of the record | the vertical merge becomes `vMergeOrig` of the record | | Numbering inserted | marker goes | the marker goes; `w:numPr` stays (observed in Word 365, 16.0.20527) | | Numbering change | the record goes | the record goes (it holds only a description of the old numbering) | - **Joining paragraphs.** The paragraph mark carries the paragraph's properties (§17.3.1.26), so when a mark goes the paragraph is joined to the next one, and the next paragraph's properties apply to the joined paragraph. A mark that cannot go is only unmarked: the last paragraph of a container, a paragraph before a table, or a paragraph that holds a section break. An emptied inserted paragraph at the end of a container is removed. - **What a property change does not cover.** `w:rPrChange` in a paragraph-mark `w:rPr` leaves the revision markers there; `w:pPrChange` leaves the mark's `w:rPr` and `w:sectPr`; `w:sectPrChange` leaves the header and footer references; row and cell changes leave their revision markers. These are separate revisions. - **Empty containers** left by a resolved revision (`w:rPr`, `w:pPr`, `w:trPr`, `w:tcPr` with nothing in them) are removed. - **Rows and tables.** Removing the last row removes the table; a cell left without a paragraph gets an empty one (§17.4.66). Removing the last cell removes the row. - **Range markers** of moves (and of custom XML) are removed with their content. After accept-all or reject-all any that remain are removed too. - **Order.** Revisions of one part are resolved from the end of the part to its start, so content moved by one revision never holds a revision that is still to be resolved. Revisions nested in each other (another author's deletion inside an insertion) resolve independently: accepting only the outer one leaves the inner one in place. - **One revision, a set, an author, a range, all.** Accepting or rejecting works on any selection taken from the current list. After any change the list is read again, and revisions taken earlier are refused (`ArgumentException`). - **What is untouched.** Nothing outside the resolved revisions changes: unknown elements, `mc:AlternateContent`, `extLst`, prefixes and attribute order stay as they were, and a part without revisions is not modified. A revision that cannot be resolved because its markup is not what the schema allows stays in place and is reported (`docx.revision-unresolvable`). ## Text views `RevisionSet.GetText` returns the text of a part as if every revision were accepted or every one rejected, without changing it: one line per paragraph, `\t` for tabs, `\n` for line breaks. A paragraph whose mark is removed in the view has no line end of its own, so it runs on into the next. Rows and cells that do not exist in the view are skipped. The view and the text after `AcceptAll`/`RejectAll` are the same by construction and are checked equal over the test corpus. Text boxes and drawings are not read. ## Authoring `Document.Revisions.CreateAuthor(name, date)` (or `RevisionSet.CreateAuthor`) gives a `RevisionAuthor`: the way a comparison tool or a program writes its edits as tracked changes. Everything it writes resolves cleanly: after `RejectAll` the part's text and structure are the original, and after `AcceptAll` the text is what the edits describe. It works on any part of the set: the body, headers, footers, footnotes, endnotes and comments. - **Ids.** `CreateAuthor(name, date)` numbers from one above the largest `w:id` in the parts (revisions, bookmarks, comment ranges, move ranges). `CreateAuthor(name, date, ids)` takes a `RevisionIds`: `StartingAt(n)` counts up from `n`; `FromCallback(next)` asks a function for every id. One source can serve several authors on one set. An id that is negative or already used throws `RevisionIdConflictException` (`docx.revision-id-conflict`); runs the call had already split are not put back, so treat the document as spoiled. The order in which ids are taken is fixed for each operation, so the same edits on the same document write the same ids. - **Dates and the clock.** The date is the one given (UTC, to the second) or none. The library never reads the clock, and the output depends only on the document and the calls: the same calls on the same bytes write the same bytes. - **Positions** are UTF-16 offsets into a paragraph's current text: the text with every revision accepted (deleted and moved-away text does not count; inserted text does). A tab or a line break is one character; field instructions and drawings are none. - **Two levels.** Every operation takes a `MarkupNode` (the low-level path, with property changes as XML fragments in the part's own prefix) and the document model's own `Paragraph`, `Run`, `Table`, `TableRow`, `TableCell` (with typed property changes). The model overloads do what the markup ones do; `InsertParagraph` and `InsertTable` return the new object. - **Text:** `InsertText`, `DeleteText`, `ReplaceText` (deletion then insertion; the new text takes the formatting of the first replaced character), `MoveText`. Runs are split at the edges of the edit and keep their properties; inserted text takes the properties of the run before it. Deleting text the same author inserted removes it outright; deleting another author's inserted text puts a deletion inside their insertion. Inserting in the middle of another author's insertion splits their wrapper. - **Moves.** `MoveText` and `MoveParagraph` write `w:moveFrom` and `w:moveTo` with `w:moveFromRangeStart/End` and `w:moveToRangeStart/End`. The two ends are paired by name: `move` by default, or the `moveName` the caller gives (refused when another move in the parts already uses it). Moved-away text stays `w:t`. - **Paragraphs and tables:** `InsertParagraph` marks the existing paragraph's mark as inserted and gives the new paragraph the old paragraph's properties (what Word writes when Enter is pressed at the end of a paragraph), so a reject restores the original properties wherever the paragraph is. `DeleteParagraph` deletes its content and mark. `MoveParagraph` moves it after another paragraph. `InsertRow(table, index, texts)`, `MarkRowInserted`, `MarkRowDeleted`, `InsertTable(after, cells)`, `DeleteTable`, `MarkCellInserted`, `MarkCellDeleted` and `MergeCells` write table revisions; a table inserted at the end of a cell is followed by an inserted empty paragraph (a cell must end with one), which a reject leaves as one empty paragraph. - **Formatting, typed.** `RunPropertiesDelta` (style, font, size in points, bold, italic, strike, caps, underline, colour, highlight), `ParagraphPropertiesDelta` (style, alignment, left/right/first-line/hanging indents, spacing before/after, line spacing as a multiple, keep-with-next, keep-lines, page-break-before, list id and level), `TablePropertiesDelta`, `RowPropertiesDelta` and `CellPropertiesDelta` set what is not null, keep everything else (including attributes of `w:ind` and `w:spacing` the delta does not name), and put new elements where the schema's order puts them. `FormatText`, `ChangeRunProperties`, `ChangeParagraphProperties`, `ChangeTableProperties`, `ChangeTableRowProperties` and `ChangeTableCellProperties` take them and record the old properties as `w:rPrChange`, `w:pPrChange`, `w:tblPrChange`, `w:trPrChange` or `w:tcPrChange`. A pending change keeps its first recorded original. A numbering change is a paragraph property change (`NumberingId`; 0 takes the paragraph out of its list). Invalid values (a colour that is not six hex digits, a size that is not a multiple of half a point, a list level above 8) throw `ArgumentException` before anything changes. Section properties and table grids take XML fragments only. - **Example.** A comparison tool recording that one word changed, one paragraph was added and a heading became centred: ```csharp using Document doc = Document.Open("before.docx"); RevisionAuthor au = doc.Revisions.CreateAuthor("Compare", new DateTimeOffset(2025, 3, 4, 5, 6, 7, TimeSpan.Zero), RevisionIds.StartingAt(1000)); List p = doc.Body.Paragraphs().ToList(); au.ReplaceText(p[2], 14, 5, "revised"); Paragraph added = au.InsertParagraph(p[2], "A new paragraph after it."); au.ChangeParagraphProperties(p[0], new ParagraphPropertiesDelta { Alignment = ParagraphAlignment.Center }); au.FormatText(added, 0, 1, new RunPropertiesDelta { Bold = true }); doc.SaveAs("after.docx"); // accepting every change gives the new text, rejecting gives before.docx's ``` ## Observed in Word Checked with Word 365 (16.0.20527) accepting and rejecting every revision of generated files: text, paragraph formatting, list labels and table cells agree with ours for every operation above, in the body, a footnote, a header and a comment. One difference: rejecting a `w:tblPrChange` through the object model removes the record but leaves the new table properties in place (and, when row or cell changes are present as well, may split the table at the first row). Our reject restores the recorded properties, as §17.13.5 describes; row and cell property changes behave alike in both. ## Not covered - Revisions inside text boxes and drawing canvases are found (they are elements in the part) and resolved, but the views do not read text boxes. - **`mc:AlternateContent`.** Only the branch a reader takes (ECMA-376 Part 3 §9.3: the first `mc:Choice` whose required namespaces are understood, else `mc:Fallback`) is listed, so a DrawingML text box with a VML copy counts once, as Word counts it. A revision in the other branch is its copy: accepting or rejecting the listed revision resolves the copy with it, and `AcceptAll`/`RejectAll` resolve every copy. Copies are paired with the listed revisions by kind and order inside the same `mc:AlternateContent`. The copy is brought up to date by `Document.CommitChanges`, which writes the exposed branch into the other. - **Property changes and paragraph marks.** Rejecting a `w:pPrChange` keeps the other revisions of the same `w:pPr` (the paragraph mark's `w:ins`/`w:del`, a row's or cell's marks) as they are, so they can still be resolved in the same call. `InsertParagraph` on a paragraph with a pending `w:pPrChange` gives the new paragraph a copy of it (a new id), so rejecting everything restores the original properties. - Tracked changes outside WordprocessingML (DrawingML, SpreadsheetML change tracking) belong to their own models. - Word's choice when two paragraphs join and both hold properties is taken from the paragraph mark rule above; whether Word keeps the first paragraph's properties when the first paragraph still has text is an open question for the Office oracle. - **Namespace declarations when markup moves.** Joining paragraphs, `InsertParagraph`, moves, unwrapping an insertion and copying a text box's story into its `mc:Fallback` copy rewrite markup under another parent. A prefix the moved markup uses that is declared above it but not on the part's root element (for example `xmlns:r` on the old `w:p`) is declared again on the moved element, so no prefix is ever written unbound. When every prefix is on the root, the bytes are the markup's own, unchanged. A prefix that is bound nowhere in the source is a defect of the source markup; the edit then fails with the existing typed outcome (`ArgumentException` when authoring, `docx.revision-unresolvable` when resolving) and leaves the revision in place. - **Wrappers emptied by resolving.** Observed in Word 365 (16.0.20527) by rejecting the only insertion in, or accepting the only deletion in, a wrapper: - `w:smartTag` and `w:customXml` (with their `w:smartTagPr`/`w:customXmlPr`) are removed, and so are such wrappers that emptying them leaves empty in turn. `w:bdo` and `w:dir` follow the same rule here (not probed). - `w:hyperlink` stays as an empty element. - `w:sdt` (inline or block) stays. Word also marks it `w:showingPlcHdr` and fills `w:sdtContent` with a placeholder run of spaces; OneDocument leaves the content empty and does not invent a placeholder, so the text of that content control differs from Word's by those spaces. - `w:fldSimple`: Word rewrites it as a complex field and recalculates its result. OneDocument keeps the `w:fldSimple` with no result run; cached results stay authoritative and nothing recalculates on open. - A block-level content control whose only paragraph had an inserted mark: Word joins the remaining content into the paragraph after the control; OneDocument leaves an empty paragraph in the control. # Word binary files (DOC) `RecursiveAI.OneDocument.Doc.Binary` (package `.Documents`) reads Word 97, 2000, 2002, 2003, 2007, 2010 and 2013 binary documents, Word 6.0 and Word 95 documents, and files that were saved incrementally ("fast saved"). Reading only: DOC writing is not planned for 1.0. A DOC opened with `Document.Open`, `Office.Open` or `Office.Convert` is converted into the WordprocessingML model (an in-memory package that saves as DOCX); `Office.ExtractText` reads the text without building the model. Specification: [MS-DOC] (Word Binary File Format), with [MS-ODRAW] for pictures and [MS-OSHARED] for number formats. Word 6.0 and Word 95 files are outside [MS-DOC]; see "Word 6/95" below for what was settled from real files. ## Layers | Type | Layer | |---|---| | `DocFile` | The compound file's `WordDocument`, `0Table`/`1Table` and `Data` streams read into memory; the FIB; the piece table; the whole text and the character range of each story. | | `DocFib` | The File Information Block: version, flags, story lengths and the (offset, size) pairs ([MS-DOC] 2.5). Pairs are read up to the count the file declares. | | `PieceTable`, `Piece` | `Clx`, `PlcPcd`, `Pcd`, `Prc` ([MS-DOC] 2.9.38, 2.8.35): where each run of characters lives, 8-bit or UTF-16, in any file order. | | `PropertyRuns` | All character or paragraph formatted disk pages (`ChpxFkp`, `PapxFkp`) behind the bin tables, flattened and sorted. | | `Prl`, `SprmReader`, `SprmNames` | Property modifiers with operands kept as bytes; unknown modifiers are carried, named when [MS-DOC] 2.6 lists them. `SprmTable.g.cs` is generated from the specification by `tools/DocSprmTable/generate.py`. | | `DocReader` | Property lookup per character position (run plus the piece's `Prm`), the paragraph walk (paragraph, cell and row marks, table depth), `Plc` arrays, section ends. | | `DocStyleSheet`, `DocFontTable`, `DocLists` | `STSH`, `SttbfFfn`, `PlfLst`/`PlfLfo`. | | `DocTextExtractor` | Text without the model. | The conversion (`DocConverter`, internal) writes WordprocessingML parts from these. All of it is bounded by the `OpenContext` limits: stream sizes by `MaxPartSize`, the text by `MaxAllocationBytes`, every length and offset checked before use. ## What is read - **Text**: all stories (main, footnotes, headers and footers, comments, endnotes, text boxes, header text boxes), 8-bit text with the `FcCompressed` mapping of 0x82 to 0x9F, UTF-16 text, complex (fast-saved) piece tables. - **Formatting**: character, paragraph, table and section modifiers through the modifier tables of [MS-DOC] 2.6 (the modifiers that are mapped are listed in the source of `RunFormat`, `ParaFormat`, `TableRowFormat`, `SectionFormat`); toggles with "same as style" and "opposite of style" are resolved against the style chain; `sprmPHugePapx` is followed into the Data stream; a piece's `Prm` (simple and complex) is applied. - **Styles** with inheritance, next style and aliases; **fonts**; **lists** (list definitions, overrides, level overrides, bullets and number formats); **fields** as complex fields (instruction kept, result kept); **bookmarks**; **footnotes and endnotes** with separators; **comments** with author and initials (anchored at the reference mark); **headers and footers** per section and kind (even, odd, first); **sections** (page size, margins, columns, orientation, borders, line numbers, page numbering, vertical alignment, document grid); **tables** (cell edges, merges, borders, shading, row height, header rows, nesting; one grid for the union of all rows' cell edges); **tracked changes** (insertions and deletions with author and date, on runs and paragraph marks); **pictures** (inline pictures from the Data stream, floating pictures from the drawing layer with position, wrapping and z-order; EMF, WMF, JPEG, PNG, DIB, TIFF) ; **document settings** (default tab stop, odd/even headers, revision tracking). ## What is lost Every loss is a `loss.doc-*` entry in `Document.ImportLoss`, and where a location exists a defect. | Code | Meaning | |---|---| | `loss.doc-sprms` | Property modifiers with no counterpart (count); each kind is one `doc.sprm-not-mapped` defect with the stream offset of the first occurrence. | | `loss.doc-text-boxes` | Text boxes: their text is appended after the body as paragraphs; position and frame are not kept. | | `loss.doc-header-text-boxes` | Text boxes in headers and footers are dropped. | | `loss.doc-drawing-shapes` | Shapes that are not pictures (lines, rectangles, WordArt, groups) are dropped. | | `loss.doc-ole-objects` | Objects whose storage could not be carried (missing from the `ObjectPool`, a link with no readable moniker, or inside tracked changes) keep only their preview picture. Objects that could be carried are not counted: see `ole.md`. | | `loss.doc-form-fields` | Form fields stay plain fields; check box state and drop-down lists are dropped. | | `loss.doc-table-merge-split` | `sprmTMerge`/`sprmTSplit` row edits. | | `loss.doc-vba` | A `Macros` storage was present. | | `loss.doc-properties` | Properties of the summary information streams with no counterpart in the package (count), each named in the message: the thumbnail, the VBA signature, hyperlink lists, custom property types without an equivalent, unknown identifiers. The rest become core, extended and custom properties: see `ole.md`. | | `loss.doc-word6-formatting` | Word 6/95 formatting. | ## Text extraction `Office.ExtractText` reads the piece table, the paragraph and character runs (for table structure and tracked deletions) and the story tables, and nothing else. Output: one line per paragraph; a table row on one line with the cells separated by a tab and the paragraphs of a cell by a space (nested tables flatten into their cell); a field gives its result and never its instruction; tracked deletions are left out (a deleted paragraph mark joins the paragraph to the next); then headers and footers (empty ones skipped), footnotes, endnotes and comments, each part set off by a blank line; then the text boxes. Footnote, comment and picture characters give no text. Differences from the OOXML extractor, none about words: the order of headers and text boxes relative to the body. ## Word 6/95 [MS-DOC] covers Word 97 onwards. A Word 6.0 or Word 95 file (`wIdent` 0xA5DC, `nFib` 101 or 104) is read with these rules, **settled by reading the headers of the 34 Word 6.0 and Word 95 files in the govdocs1 set** (4 with `nFib` 101, 30 with 104) (the black-box documents; nothing else was consulted): - the header is one fixed block: `fcMin` and `fcMac` at 0x18 and 0x1C bracket the 8-bit text (the text exceeds the sum of the story lengths by one character in 28 of the 34 files and equals it in 6); - the story lengths are 32-bit values from 0x34: main text, footnotes, headers, macros, comments, endnotes, text boxes, header text boxes; - the offset pairs start at 0x58, in the order of `FibRgFcLcb97`; - there is no table stream: structures that live there in Word 97 are in the `WordDocument` stream; - a document that was saved in full has no piece table (the text is one 8-bit run); a fast-saved one has a `Clx` whose `Pcd.fc` values are plain byte offsets (no compression flag); - the 8-bit text is in the character set at 0x14 (0 is ANSI); other sets are read as Windows-1252 with a `doc.codepage-unsupported` defect; - paragraphs end at 0x0D, table cells at 0x07, the row end is a cell mark that follows another with nothing between. Word 6/95 character, paragraph and section formatting uses a different, one-byte modifier numbering that [MS-DOC] does not describe and that was not settled; it is reported as `loss.doc-word6-formatting`. Styles, lists and pictures of those files are likewise not read. Word 1.0 and 2.0 files (a flat file starting `FE 37 00 23`, no compound file) and files that are not compound files at all are not recognised: `format.unrecognised`. ## Encryption A file with `fEncrypted` set is decrypted with `OpenContext.Password` (XOR, RC4, RC4 CryptoAPI), or raises `OfficeEncryptedException`; see `encryption.md`. Password-protected Word 6/95 files raise `OfficeUnsupportedException`. ## Questions Office decides Committed as unanswered cases for the 1.0 gate (`testdata/doc/word-conversion-cases.md`): how Word's own conversion of a DOC to DOCX differs from this one on tables with differing row edges, on lists linked to styles, on footnote/comment numbering and on text boxes. # RTF (Rich Text Format 1.9.1) RTF is read into and written from the word-processing hub (`Document`). The format has no part structure to keep byte-for-byte, so the lossless layer is a group tree of the file, and the hub is the WordprocessingML model: a document read from RTF is a WordprocessingML package in memory (`Document.Format` is `OfficeFormat.Rtf`), and everything the hub does works on it. Conversions return loss reports. ## Layers | Layer | Namespace / package | What it is | |---|---|---| | Tokenizer | `RecursiveAI.OneDocument.Rtf` (core) `RtfReader` | Forward-only tokens over a byte span: groups, control words (with the one delimiting space), control symbols, `\'hh`, text runs, `\binN` data. Offsets on every token. Never throws. | | Group tree | `RtfTree`, `RtfNode` (core) | The whole file as nested groups with source offsets; text and binary nodes point into the source bytes. `RtfTree.SourceOf(node)` is the exact bytes of any group, so unknown destinations are kept as bytes. | | Writer | `RtfWriter` (core) | Groups, words, escaped text (7-bit ASCII: `\'hh` for Windows-1252, `\uN?` otherwise), `\bin`, picture hex. | | Mail layer | `RtfHtmlConverter`, `RtfTextWriter`, `RtfEncapsulator`, `RtfDeencapsulator`, `Compression.*` (core) | The mail-body subset: RTF to HTML and text, HTML and text encapsulation in RTF, compressed RTF. Kept as the mail library had it. | | Hub conversion | `.Documents` (internal `RtfImporter`, `RtfExporter`) | RTF to WordprocessingML and back, reached through `Document.Open`, `SaveAs("x.rtf")`, `Office.ExtractText` and `Office.Convert`. | `Document.Open` finds RTF by its `{\rtf` signature, never by extension. `Document.ImportLoss` is the loss report of the read, `Document.ExportLoss` that of the last write to RTF. ## Round trip A document opened from RTF and saved as RTF with no edit writes **the bytes it was read from**. After any edit (the hub notes an edit when a part is changed), a save writes the exporter's RTF: the hub's content in the word processor's own RTF dialect. Header-level destinations this library does not use (`\*\generator`, `\*\rsidtbl`, `\*\userprops` and the like) are kept as bytes and written back after the `\info` group. ## Reading Every control word of RTF 1.9.1 that has a WordprocessingML counterpart is read; the rest is counted in the report or ignored as a layout hint Word recomputes. | RTF | Becomes | |---|---| | `\fonttbl` (families, `\fcharset`, `\cpg`, `\falt`) | `fontTable.xml`; the font's charset chooses the code page for `\'hh` bytes, so double-byte text decodes | | `\colortbl` | colours as `w:color`, `w:shd` fills, `w:highlight` (the sixteen named colours) | | `\stylesheet` (`\s`, `\*\cs`, `\*\ts` read, `\sbasedon`, `\snext`) | `styles.xml`; paragraph and character styles with their properties; table styles are not carried; `\s0` is the default style | | character formatting | `w:rPr` in schema order: bold, italic, underline styles, strike, caps, small caps, hidden, outline, shadow, emboss, imprint, super/subscript, `\up`/`\dn`, `\expnd`, `\charscalex`, `\kerning`, `\fs`, `\cf`, `\highlight`, `\cb`, `\lang`, `\cs`, `\rtlch` | | `\u`, `\uc`, `\upr`/`\ud` | Unicode characters with the ANSI fallback skipped; the `\ud` copy is read | | paragraph formatting | `w:pPr`: alignment, indents, spacing (`\sl` with `\slmult`), keep/page-break/widow control, borders, shading, tab stops with leaders, outline level, bidi | | `\listtable`, `\listoverridetable`, `\ls`, `\ilvl` | `numbering.xml`: one abstract list per RTF list, one `w:num` per override, level format, text (`\leveltext` with its level placeholders), start value, justification, follow, indents | | tables (`\trowd`, `\cellx`, `\cell`, `\row`) | `w:tbl` on a grid made of every cell edge any row uses; `gridSpan`, `gridBefore`/`gridAfter`, vertical and horizontal merges, borders, shading, cell and row properties, header rows, row height | | nested tables (`\itap`, `\nestcell`, `\nestrow`, `\*\nesttableprops`) | nested `w:tbl` in the cell | | sections (`\sectd`, `\sect`, `\sbk*`, `\pgwsxn`…, `\cols`, `\titlepg`, `\pgn*`) | `w:sectPr` on the last paragraph of each section and in the body | | headers and footers (`\header`, `\headerl`, `\headerr`, `\headerf`, the footers) | `header`/`footer` parts referenced by type (default, even, first) from the section | | `\footnote` (and `\ftnalt` for endnotes) with `\chftn` | `footnotes.xml`/`endnotes.xml` with separators, reference runs and `footnoteRef` | | `\field`, `\fldinst`, `\fldrslt`, `\flddirty`, `\fldlock`, `\chpgn`, `\chdate`, `\chtime` | complex fields (begin, instruction, separate, result, end); fields may span paragraphs | | `\*\bkmkstart`, `\*\bkmkend` | bookmarks | | `\*\atnid`, `\*\atnauthor`, `\*\annotation`, `\*\atrfstart`/`atrfend`, `\atndate` | comments with their ranges, author, initials and date | | `\revtbl`, `\revised`, `\deleted`, `\revauth`, `\revdttm` and the `del` forms | `w:ins`, `w:del` with `w:delText`, author and date | | `\pict` with `\pngblip`, `\jpegblip`, `\emfblip`, `\wmetafile`, `\dibitmap`; `\shppict`/`\nonshppict` | media parts and inline drawings. A metafile without the placeable header gets one from `\picw`/`\pich`; a DIB gets a bitmap file header. Sizes: `\picwgoal`, scale, crops (`a:srcRect`). `\shppict` is read and `\nonshppict` skipped when both are present | | `\shp` | a picture shape (`pib`) as an inline picture; a text shape (`\shptxt`) as ordinary paragraphs at the place of the shape | | `\object` | the picture of `\result` (OLE data is never read) | | `\mmath` | Office Math: the groups are OMML element for element (`{\mf …}` is `m:f`, `{\mchr x}` is ``, `{\mr text}` is `m:r`) | | `\info` | core and extended document properties (title, subject, author, keywords, comment, operator, category, company, manager, creation and revision time) | | `\paperw`…, `\deftab`, `\facingp`, `\margmirror`, `\revisions`, `\deff`, `\deflang` | page setup of the first section, settings, document defaults | | `\pntext` (legacy bullets and numbers) | the label stays as text; the `\pn` definition is dropped | Unknown destinations (`{\*\name …}`) and unknown control words are skipped. At the top of the file an unknown destination is also kept as bytes for writing back. ### Loss report (reading), codes | Code | When | |---|---| | `loss.rtf-unknown-destination` | an extension destination the reader does not know (`\*\…`), other than the ones that only hold editing state (`\generator`, `\rsidtbl`, `\datastore`…) | | `loss.rtf-shape` | a drawing shape: its picture or text is kept, its position, size, wrapping and geometry are not | | `loss.rtf-text-box` | the text of a shape read as ordinary paragraphs | | `loss.rtf-drawing-object` | Word 97 `\do` drawing objects | | `loss.rtf-ole-object` | an embedded object: only its preview picture is read | | `loss.rtf-picture` | a picture that could not be carried (damaged, unsupported format, too large) | | `loss.rtf-frames` | positioned paragraphs and drop caps (`\absw`, `\posx`, `\dropcapli`…) read as ordinary paragraphs | ## Writing The exporter writes `\rtf1\ansi\ansicpg1252\uc1\deff0`, the font table (family and charset), colour table, style sheet (document defaults go into the default style, because RTF has no separate defaults), `\listtable` and `\listoverridetable`, `\revtbl` when there are tracked changes, `\info`, kept header destinations, page setup, then the sections. Each paragraph starts `\pard\plain` with its style and direct properties, each formatted run is a group, characters beyond Windows-1252 are `\uN` with a `?` fallback. Tables are written with their row definitions (top-level rows before their cells, nested rows in `\*\nesttableprops` after them), cell spans as `\clmgf`/`\clmrg` cell groups, vertical merges, borders and shading. Hyperlinks become `HYPERLINK` fields. Comments carry their ranges. Footnotes and endnotes are written inline at their reference. A header or footer a section shares with the one before is written once, because a section without its own shows the previous one's. Pictures (PNG, JPEG, EMF, WMF, BMP) are written as hex; a text box is written as a `\shp` with a `\shptxt` part. ### Loss report (writing), codes | Code | When | |---|---| | `loss.rtf-shape` | drawing shapes, charts and diagrams other than pictures and text boxes | | `loss.rtf-shape-position` | a text box's anchor, offsets and wrapping | | `loss.rtf-floating-picture` | a floating picture is written inline | | `loss.rtf-picture-format` | a picture in a format RTF cannot hold (GIF, TIFF, SVG, WebP…) | | `loss.rtf-picture` | a linked or missing picture | | `loss.rtf-ole-object` | an embedded object is written as its preview only | | `loss.rtf-equation` | an equation is written as its linear text | | `loss.rtf-content-control` | a content control is written as its content | | `loss.rtf-table-style` | table styles | | `loss.rtf-run-effect` | text effects, emphasis marks, fit-text and character borders | | `loss.rtf-frames` | paragraph frames | ## Defects Codes are in `defects.md` (`rtf.*`, kind `Record`, located by byte offset; limits are kind `Limit`). Reading never throws anything but a typed `OfficeException`: damage the reader repairs is a defect, input that is not RTF raises `rtf.not-rtf`, and an unexpected failure while converting is wrapped as `rtf.damaged`. Limits: the file must fit `OfficeLimits.MaxPartSize`; group nesting is bounded by `MaxRecursionDepth` (at most 1000); the tree is bounded by `MaxRecordCount` nodes; pictures count against `MaxPartSize` each and `MaxAllocationBytes` together. ## Where the specification leaves a choice - **Picture scale and crop.** `\picscalex`/`y` apply to the picture after cropping, and the crops are twips of the `\picwgoal`/`\pichgoal` size. This is the way LibreOffice writes them (checked on its output for a centred crop); Word is the reference and decides at the milestone check. - **`\sectd` and headers.** `\sectd` resets the section properties but not the headers and footers collected for the section; a new section starts with none, which is how WordprocessingML inherits them. - **`\uc` skipping.** After `\uN`, the next `\ucN` characters are skipped when they are `\'hh` escapes or text bytes; control words are never skipped. - **Footnote marks.** In RTF the body's `\chftn` and the note's own `\chftn` are both marks. The body's is replaced by the reference run the note group produces; the note's becomes `footnoteRef`. - **Charset 0 fonts** use the document's `\ansicpg`. ## Checks Unit tests (`RtfTests`, the lifted mail-layer tests), `fuzz` targets `rtftree` and `rtf`, and `tools/RtfProbe` (bulk read and write, text comparison with another extractor, and `oracle.sh` which runs LibreOffice as a black box over the same documents). Files that settle how the desktop word processor opens RTF are generated by `RtfProbe write` and listed in `testdata/rtf/word-open/`; they are answered at the milestone check. # Office 2003 XML (WordprocessingML 2003, SpreadsheetML 2003) Both formats are read-only. A file opens as a `Document` or a `Workbook` by **conversion**: the reader turns the 2003 file into an in-memory OOXML package (DOCX or XLSX) on the markup layer and the hub wraps that package like any other. What the hub then holds saves as DOCX or XLSX, extracts as text and converts as any OOXML file does. `Document.ImportLoss` and `Workbook.ImportLoss` list what did not map (null for a file that was opened natively). Writing the 2003 formats is not supported. ```csharp using Document doc = Document.Open("old.xml"); // sniffed: Word 2003 XML using Workbook wb = Workbook.Open("old.xml"); // sniffed: Excel 2003 XML Office.Convert("old.xml", "new.docx"); // or .xlsx LossReport? lost = doc.ImportLoss; ``` The static entry points are `WordMl2003.ToPackage/Open` (`RecursiveAI.OneDocument.Documents.Xml2003`) and `SpreadsheetMl2003.ToPackage/Open` (`RecursiveAI.OneDocument.Spreadsheets.Xml2003`); they take an `IByteSource`, an `OpenContext` and an optional `ILossSink`. ## Identification By content, never the extension (`FormatSniffer`): the `mso-application` processing instruction (`Word.Document`, `Excel.Sheet`) or the root namespace (`http://schemas.microsoft.com/office/word/2003/wordml`, `urn:schemas-microsoft-com:office:spreadsheet`), ahead of generic XML. The files are `.xml` (Word's "Word 2003 XML Document", Excel's "XML Spreadsheet 2003") and sometimes `.doc` or `.xls` by name. ## Reading rules - The file is parsed once with the markup tokenizer (no `System.Xml`). DTDs and entity declarations are refused; nesting, attribute counts and part size obey `OfficeLimits`. Not well-formed XML is `xml.*`; a root that is not the format's is `wml2003.not-wordml` or `sml2003.not-spreadsheet`. Everything else is repaired and reported (`defects.md`); the reader raises nothing but `OfficeException`. - Order in the output follows the ECMA-376 schema sequences, not the order in the 2003 file: property lists (paragraph, run, table, row, cell, section, style, list level) are read into a set and written in schema order. A property that appears twice keeps its last occurrence. - Hyperlink targets become external relationships. Nothing is fetched, linked pictures are not followed, and data in `w:binData` is decoded but never executed. - Names carry hyphens in 2003 (`w:first-line`, `w:h-ansi`, `w:line-rule`) and camel case in 2006. Attribute and element names convert by that rule, with `w:fareast` → `eastAsia`. Hyphenated enumeration values (`dash-long`, `next-page`, `at-least`) convert the same way. Fractional measures (LibreOffice writes `4320.5401`) round to whole numbers; empty attributes are left out. ## Word | 2003 | Becomes | |---|---| | `o:DocumentProperties` | `docProps/core.xml` (title, subject, author, keywords, description, last author, revision, created, saved, category) and `docProps/app.xml` (company, manager, pages, words, characters, lines, paragraphs, characters with spaces, total time) | | `o:CustomDocumentProperties` | `docProps/custom.xml` (`dt:dt` string, boolean, integer, float) | | `w:fonts/w:defaultFonts` | `w:docDefaults/w:rPrDefault/w:rFonts` | | `w:styles/w:style` | `word/styles.xml`; `wx:uiName` is a hint and ignored (`w:name` is kept); latent styles are dropped | | `w:lists` (`w:listDef`, `w:list`) | `word/numbering.xml` (`w:abstractNum`, `w:num`, level overrides); `w:nfc` codes map to number formats; `w:listPr` in a paragraph becomes `w:numPr` | | `w:docPr` | `word/settings.xml`: zoom, track revisions, default tab stop, character spacing control; even-page headers set `w:evenAndOddHeaders` | | `wx:sect`, `wx:sub-section` | read through; a section's `w:sectPr` goes on the last paragraph before it, the last section's closes the body | | `w:hdr`, `w:ftr` (in `w:sectPr`) | `word/headerN.xml`, `word/footerN.xml` with `odd` → default, `even`, `first` | | `w:p`, `w:r`, `w:t`, `w:tab`, `w:br`, `w:sym`, … | the same elements; a run nested in a run (LibreOffice inside a hyperlink) becomes a sibling | | `w:tbl` | tables with grid, `w:hmerge` → `w:gridSpan`, `w:vmerge` → `w:vMerge`; a missing grid is made from the widest row; a cell always ends with a paragraph | | `w:fldSimple`, `w:fldChar` | unchanged | | `w:hlink` | `w:hyperlink` (`w:dest` → relationship, `w:bookmark` → `w:anchor`) | | `aml:annotation` bookmark start/end | `w:bookmarkStart/End` | | `aml:annotation` `Word.Comment*` | `word/comments.xml`, `w:commentRangeStart/End`, `w:commentReference` | | `aml:annotation` `Word.Insertion`, `Word.Deletion` | `w:ins`, `w:del` (text becomes `w:delText`) with author and date | | `w:footnote`, `w:endnote` (inline in a run) | `word/footnotes.xml`, `word/endnotes.xml` with separators, and the reference | | `w:pict` with `v:imagedata` and `w:binData` | a DrawingML inline picture (`word/media/imageN.*`; the type is taken from the bytes, the size from the VML `style`) | | `w:pict` with `v:textbox` | a VML text box with converted content | | `w:smartTag`, `w:customXml`, `w:sdt` | removed, text kept | Dropped, with `loss.*` codes: `loss.unmapped-markup.` (any element without a counterpart, one line per element name), `loss.shapes` (VML shapes with neither picture nor text), `loss.pictures`, `loss.ole-objects` (reduced to the preview picture), `loss.smart-tags`, `loss.custom-xml`, `loss.formatting-revisions` (tracked formatting changes), `loss.comments` (a comment anchored between paragraphs). Proofing marks and permission ranges are dropped silently: they describe the editing session, not the document. ## Excel | 2003 | Becomes | |---|---| | `Worksheet`, `Table`, `Row`, `Cell` | sheets, rows, cells. `ss:Index` places an item (a backward index follows the previous item, `sml2003.bad-index`); `ss:Span` repeats a column or row; `ss:MergeAcross`/`MergeDown` merge | | `Data` `ss:Type` | `Number`; `String` (inline string); `DateTime` (ISO 8601 → serial in the workbook's date system; a time alone on 1899-12-31 is a fraction of a day); `Boolean`; `Error` | | `ss:Data` with HTML (`B`, `I`, `U`, `S`, `Sub`, `Sup`, `Font`) | rich inline string runs | | `ss:Formula` (R1C1) | A1 text through the public formula parser and printer; the cached `Data` stays as the cached result; nothing is recalculated on open. `ss:ArrayRange` gives an array formula. Text the parser rejects is `sml2003.bad-formula` and the value stays as a constant | | `Styles/Style` | cell formats; `ss:Parent` is resolved (a style with no parent inherits from `Default`); the number format names of the 2003 documentation (`Short Date`, `Percent`, `Currency` …) map to format codes | | `Column`, `Row` attributes | widths (points → characters, below), heights, `ss:Hidden`, `ss:StyleID` (a cell without a style takes its row's, then its column's) | | `Names`, `NamedRange`, `NamedCell` | defined names (workbook and sheet level); `Print_Area`, `Print_Titles`, `Criteria`, `Extract`, `_FilterDatabase` become the `_xlnm.` names | | `Comment` | `xl/commentsN.xml` and a legacy drawing so the note shows | | `ss:HRef` | a hyperlink (`#Sheet!A1` is internal) | | `x:ConditionalFormatting` | `cellIs` and `expression` rules with a differential format from the CSS in `Format Style` | | `x:DataValidation` | whole, decimal, list, date, time, text length and custom rules | | `x:AutoFilter` | an auto filter range | | `x:WorksheetOptions` | tab selection, gridlines and headings, right-to-left, zoom, frozen panes, protection, page margins, orientation, scale, paper size, centering, header and footer | | `x:ExcelWorkbook` | date system (`Date1904`), active sheet, structure protection | | `o:DocumentProperties` | core and app properties | Deviations and choices: - **Column widths.** 2003 stores points; XLSX stores characters of the default font's digit. The reader assumes a 7 pixel digit and 5 pixels of padding at 96 dpi: `(points × 96 / 72 − 5) / 7`, rounded to two places (48 points is 8.43 characters). - **Duplicate style IDs.** The first definition wins (`sml2003.duplicate-style`). LibreOffice writes `Default` twice, the second time in full. - **Named styles.** A named cell style (`ss:Name` other than `Default`) becomes a plain cell format; the name is reported as `loss.cell-style-names`. - **Sheet names.** Characters `[]:*?/\` are removed, names are cut to 31 characters and made unique ignoring case (`sml2003.sheet-name`). - **OpenFormula text.** LibreOffice's "Microsoft Excel 2003 XML" export writes `of:=[.B2]+[.B3]` in `ss:Formula`. That is not R1C1 and is not read here (OpenFormula belongs to the ODF reader); each such formula is a defect and its cached value is kept. - **Not carried over**: manual page breaks (`loss.page-breaks`), sorting, pane details beyond the frozen split, printer settings. ## Checks and open questions Checked on the build host: every file the sniffer calls Word or Excel 2003 XML among the committed fixtures and 87 Word and 4 Excel files written by LibreOffice from its own corpus opens with typed defects only; their DOCX/XLSX conversions validate against the pinned ECMA-376 Transitional schemas (1,157 of 1,158 parts; the one miss is `xml:space` on a spreadsheet `t` element, which the schema does not declare and Excel itself writes). Unanswered by Office (to be answered on the oracle machine): 1. Does Word open the DOCX converted from `testdata/xml2003/word-features.xml` and `lo-word.xml` without a repair prompt, and does Word's own conversion of the 2003 file match (paragraph count, numbering labels, revisions, comment text, picture size)? 2. Does Excel open the XLSX converted from `excel-features.xml` without repair (comments with the legacy drawing, conditional formats, validation), and does Excel's own "Save as .xlsx" of the 2003 file give the same values, formats and formulas? 3. Which style wins when a style ID is defined twice, and does a cell without `ss:StyleID` take its row's or its column's style? 4. What column width does Excel give `ss:Width="64.008"` on a machine with the default font (the width formula above is an estimate)? Regenerate the conversions with `tools/Xml2003Probe convert `; `tools/xml2003/` holds the inputs LibreOffice converted for the fixtures. # HTML import and export HTML reads into the word-processing hub and the hub writes it back. Both directions report what they could not carry. ```csharp using Document doc = Document.Open("page.html"); // sniffed by content: , , , or a tag such as

,

, using Document frag = Document.FromHtml("

Title

Hello world

"); LossReport? lost = doc.ImportLoss; // scripts, handlers, unsafe links, forms, frames, external images ... Office.Convert("page.html", "page.docx"); // and "page.docx" -> "page.html" string html = doc.ToHtml(); // a page; new HtmlExportOptions { Fragment = true } for the content only doc.SaveAsHtml("out.html", new HtmlExportOptions { ImageSink = i => { File.WriteAllBytes(i.SuggestedName, i.Data.ToArray()); return i.SuggestedName; } }); ``` `Workbook.ToHtml` and `Presentation.ToHtml` (and `SaveAs("x.html")` on them) write a basic export, below. ## Import The page is parsed as a browser parses it (OneHtml: encoding from the byte-order mark or `meta`, tree construction, error recovery) and its cascade (user-agent rules, `style` elements, `style` attributes, legacy attributes such as `align`, `bgcolor`, `border`, `font`) supplies a computed style per element. The importer reads those computed values; it does not interpret CSS itself. The result is an in-memory DOCX package that the hub wraps. | HTML | Becomes | |---|---| | `h1`..`h6` | `Heading1`..`Heading6` (outline levels 0..5) with the formatting of the first heading of each level | | `p`, `div`, `section`, `li`... with inline content | paragraphs; text directly in a block is an anonymous paragraph with the block's style | | `class` | a named paragraph style per first class (`.note` becomes style `note`); the first paragraph with the class defines what the style holds, later ones carry only what differs. Classes written by the exporter (`s-`, base first) read back as those styles with their `basedOn` chain | | `b`/`strong`, `i`/`em`, `u`, `s`/`del`, `sup`, `sub`, `small-caps`, `color`, `font-size`, `font-family`, `background-color` (inline), `lang` | run properties; only what differs from the paragraph's style is written | | `text-align`, `margin-*` and `padding-left/right` of containers, `text-indent`, `line-height`, `border-*`, `background-color` on a block | paragraph alignment, indents (nested blocks add up), spacing before (adjacent margins collapse as in CSS: the larger wins), line spacing, borders, shading | | `ul`, `ol`, `type`, `start`, `list-style-type` | numbering definitions; one `w:num` per list element, so each ordered list restarts at its `start` (`startOverride`); nesting is the level; a second paragraph in an item continues it without a marker | | `table`, `thead`, `tbody`, `tfoot`, `tr`, `th`, `td`, `caption`, `col` | tables: `colspan` is `gridSpan`, `rowspan` is `vMerge`, `thead` and leading all-`th` rows are repeated header rows, widths (px, pt, %) become the grid and `tblW`, cell borders, shading, padding and vertical alignment are kept, `border` and `align` on the table too; a caption is a paragraph before the table | | `a href`, `a name`, `id` | hyperlinks (external relationships or `w:anchor`) and bookmarks. Only ids that a `#fragment` link names become bookmarks; names are made valid for Word (letters, digits, underscore, at most 40 characters) | | `img` with a `data:` URI | an inline picture (PNG, JPEG, GIF, BMP read from the header; the size comes from `width`/`height` attributes or CSS, else the picture's pixels, capped to the text width); `alt` becomes the description | | `pre`, `white-space: pre*` | text with preserved spaces, tabs and line breaks; `br` is a line break; `code`/monospace families map to Courier New | | `blockquote` | style `Quote` and the browser's indent | | `hr` | an empty paragraph with a bottom border | | `page-break-before/after`, `break-before/after` (page, left, right, always) | `pageBreakBefore` on the next paragraph | | `dir`, CSS `direction` | `bidi` on paragraphs, `rtl` on runs of right-to-left script, `jc` swapped to the logical side | | `title`, `meta name=author/description/keywords` | document properties | White space collapses as CSS says (runs of white space become one space, leading space in a block and trailing space at its end are dropped, a space between inline elements is kept once). ### Sanitising Import is `HtmlSanitization.Strict` unless told otherwise. Nothing is ever executed or fetched. Each of these is left out and reported once per occurrence in the loss report and as an `html.dropped` defect with the text position of the element: | Loss code | What | |---|---| | `loss.html-script` | `script` elements (their text is not kept) | | `loss.html-event-handler` | attributes starting `on` (never read) | | `loss.html-unsafe-link` | link targets other than `http`, `https`, `mailto`, `tel`, `ftp`, `ftps`, `sms`, relative targets and `#fragments`; the check ignores the tabs, line feeds and spaces a browser ignores inside the scheme, so `java\tscript:` is `javascript:`. The link text stays | | `loss.html-form`, `loss.html-form-control` | the `form` wrapper (its text and blocks stay) and `input`, `button`, `select`, `textarea`, `option`, `output`, `meter`, `progress` (with their content) | | `loss.html-embedded-content` | `iframe`, `frame`, `frameset`, `object`, `embed`, `applet` | | `loss.html-media` | `canvas`, `audio`, `video` | | `loss.html-svg`, `loss.html-math` | inline SVG and MathML | | `loss.html-external-image` | an `img` whose `src` is not a `data:` URI and that the resolver did not supply; its `alt` text stays | | `loss.html-image-format` / `loss.html-image-unreadable` | a picture that is not PNG, JPEG, GIF or BMP, or a `data:` URI that does not decode | | `loss.html-linked-stylesheet` | `link rel=stylesheet` (never fetched) | | `loss.html-hidden` | `display: none` and the `hidden` attribute | | `loss.html-depth`, `loss.html-table-size` | content nested deeper than `OfficeLimits.MaxXmlDepth`, or tables past 200,000 cells | `HtmlImportOptions.Sanitization = HtmlSanitization.KeepInert` keeps the raw-but-safe superset: the text of form controls (button labels, field values, option text) and the fallback text of frames and objects as plain text, and links with other schemes. It never admits scripts, handlers or `javascript:`, `vbscript:`, `data:`, `blob:` and `file:` targets, because a document cannot hold them. `HtmlImportOptions.ImageResolver` is the only way an external `img` is read: the importer hands it the `src` as written and uses the bytes it returns; it never opens a URL itself. ### Limits `OfficeLimits.MaxPartSize` bounds the input; `MaxXmlDepth` bounds element nesting (the parser stops at that many open elements with an `OfficeLimitException`) and the importer's recursion; `MaxXmlAttributeCount` bounds attributes per tag; `MaxRecordCount` bounds the cells of one sheet in the workbook export. Embedded pictures are bounded by `MaxPartSize`. The importer reads no file, no URL and no style sheet it was not given. ### Templates The template engine's `html` tag runs on this importer: the same parse, cascade and sanitising, flattened to the small vocabulary a tag can insert into an existing document (paragraphs, headings, bold, italic, underline, strike, code, links, breaks, quotes and nested lists). Tables, pictures and colours in the value are dropped and reported as `template.markup-dropped`. ## Export `Document.ToHtml` writes UTF-8 HTML5, deterministic and self-contained: | Document | HTML | |---|---| | paragraph styles and character styles | CSS classes `s-` in one `style` element; a style based on another lists both classes, base first; direct formatting is an inline `style` on the paragraph or a `span` | | `heading N` styles (by name, through `basedOn`) | `h1`..`h6` | | bold, italic, underline, strike-through, super/subscript | `strong`, `em`, `u`, `s`, `sup`, `sub`; colour, size, font, shading, caps in a `span` | | numbered paragraphs | `ol` / `ul` nested by level; `start`, `type` (`a`, `A`, `i`, `I`), bullet kind from the level text; numbering continues where it left off if the list is interrupted | | tables | `table` with `colgroup`, `thead`/`th` for repeated header rows, `colspan`, `rowspan`, borders (direct, then the table style's), shading, widths, vertical alignment, cell margins | | hyperlinks, bookmarks, `HYPERLINK` fields | `a href`, `span id`; targets keep the import allow-list | | footnotes and endnotes | a `sup` link per reference (numbered by CSS counter, so no text is added to the body) and one `section role="doc-endnotes"` ordered list at the end with back links | | pictures | `img` with `alt`, `width` and `height`; a `data:` URI, or the URL the `ImageSink` returns (the sink gets a unique name, the media type and the bytes; returning null keeps the picture inline). EMF, WMF and TIFF are not shown by browsers: their alt text is written and `loss.html-metafile` counts them | | page and section breaks | an empty `div` with `break-after: page`; the paragraph is split at an inline page break | | right-to-left paragraphs and runs, languages | `dir="rtl"`, `lang`; `html lang` from the document default | | text boxes | blocks after the paragraph they were anchored in (`loss.html-text-boxes`) | | ruby (phonetic guide) | `ruby` and `rt` | | equations | their linear text in a `span class="math"` (the text extractor's form) | | empty paragraph | `


` | The style sheet puts `white-space: pre-wrap` on paragraph elements, so the spaces and breaks of the document are exactly those of the HTML. Not written, each a loss item on `Document.ExportLoss`: `loss.html-headers-footers` (HTML has no pages), `loss.html-comments`, `loss.html-revisions` (deleted text; inserted text is written as plain text), `loss.html-equations` (plain text), `loss.html-charts`, `loss.html-drawings` (shapes, SmartArt, OLE objects without a picture), `loss.html-floating-pictures` (written in the line), `loss.html-text-boxes`, `loss.html-metafile`, `loss.html-unsafe-link`. The page this writes reads back with `Document.FromHtml` to the same text, the same headings and styles, the same lists (levels, kinds, starts), the same tables (grid, spans, merges, header rows, shading) and the same bookmarks and links. Notes come back as an ordered list at the end; the other losses above stay lost. ### Workbooks and presentations `Workbook.ToHtml` writes one `table` per visible worksheet under an `h2` with the sheet name. Values are shown as Excel shows them: numbers and dates go through the cell's number format (the shared number-format service), text as is, booleans and errors as text. Numbers are right-aligned (`td.n`). Merged ranges are `colspan`/`rowspan`. Not written: styles other than the number format, column widths, hidden sheets (`loss.html-hidden-sheets`), chart sheets, formulas (cached results are written, `loss.formulas`), drawings, charts, comments and pivot tables (`loss.embedded-objects`). `Presentation.ToHtml` writes a `section` per slide: the title placeholder as `h2`, other text shapes as paragraphs (indented by level) in shape order, tables as tables, notes in an `aside`; hidden slides carry `hidden`. Not written: positions and sizes, pictures, charts and other graphic frames, animations, transitions, masters and layouts (`loss.html-slide-layout`, `loss.html-pictures`, `loss.html-graphic-frames`, `loss.html-animations`). ## Defects `html.dropped` (one per left-out element, with its offset), `html.damaged` and `html.export-failed` (wrapped unexpected failures); limits raise `OfficeLimitException`. See `defects.md`. ## Not here MHTML and single-file archives, CSS paged media (`@page`, running headers), `float`/`position` layout, `` and `` content, Markdown (its own importer), and any rendering of HTML through OneRender's CSS engine: pages are mapped to the document model and laid out by the document layout core. ## Oracles LibreOffice opens the exported HTML (`soffice --convert-to`); Word's reading of our HTML, and Word's own HTML export of the same documents, are recorded for the desktop-Office run in `testdata/html/word-cases.md`. # Markdown CommonMark 0.31.2 with the GitHub Flavored Markdown extensions, into and out of the document hub, and out of workbooks and presentations. Markdown is the one format that is never recognised from content: the caller names it (`Document.OpenMarkdown`, `Document.ParseMarkdown`, `Document.Open(path, OfficeFormat.Markdown)`, `Office.Open(path, OfficeFormat.Markdown)`), or it is chosen by a file name's extension (`Office.Convert("a.md", "a.docx")`, `SaveAs("a.md")`). Text that merely looks like Markdown is plain text to `FormatSniffer`. ## The parser `Documents/Markdown/MarkdownParser.cs` is a block parser that works line by line as CommonMark's appendix describes (each line meets the open containers, then the block starts, then lazy continuation); `MdInlineParser.cs` does the inlines with the delimiter-run algorithm of §6.2 and the bracket stack of §6.3. Both are our own code, written against the specification text. The tree is `MdNode` (internal). Dialect switches (`MarkdownParserOptions`): tables, strike-through (one or two tildes), task lists, extended autolinks (`www.`, `http://`, `https://`, `ftp://`, e-mail, `mailto:`/`xmpp:`), footnotes (`[^1]` and `[^1]:`), raw HTML. The import uses all of them; the specification run uses `CommonMark` (all off) for the core examples. **Limits and run time.** The parser is linear in the input: - block containers nest at most `OfficeLimits.MaxRecursionDepth` deep (at least 8); a marker beyond that is read as text and reported as `markdown.too-deep`; - per line, work is bounded by that depth; - a failed search for a code-span closer, an HTML comment, processing instruction, CDATA section or declaration closer is remembered, so a text of many unclosed openers is not scanned again for each; - emphasis uses the `openers_bottom` bounds, link deactivation walks only the part of the bracket stack not already deactivated, link destinations stop at 32 nested parentheses, and an autolink scan stops at the first space or `<`; - no regular expressions are used in the parser. Nesting inside inline content (emphasis in emphasis, images in images) is not limited in the tree; every walker (importer, template converter, text of an image) stops flattening at `MaxRecursionDepth`. Character references: numeric ones are decoded in the parser; named ones go through OneHtml's table, so the whole HTML5 set is known. ## Conformance The tests fetch the published specifications when they run (the text is CC BY-SA and is not kept in the repository): | Specification | Where | Examples | Result | |---|---|---|---| | CommonMark 0.31.2 | `https://spec.commonmark.org/0.31.2/spec.json` | 652 | 652 pass | | GFM (0.29-gfm), extension sections: Tables, Task list items, Strikethrough, Autolinks, Disallowed Raw HTML | `https://github.github.com/gfm/` (the examples are read from the page) | 28 | 28 pass | Method: the tree is rendered to HTML in the shape the specifications print (`tests/OneDocument.Documents.Tests/MdHtml.cs`, which also applies the GFM tag filter), white space between tags is ignored (as the specifications' own runners do) and the two strings are compared. Set `ONEDOCUMENT_SPEC_DIR` to keep the downloaded files between runs; without network the tests skip. Failures by section go to `$ONEDOCUMENT_ARTIFACTS/commonmark-spec.txt` and `gfm-spec.txt`. The GFM core sections are not run against the GFM page: that page is CommonMark 0.29, and the CommonMark 0.31.2 run covers the core. ## Import: Markdown to the hub `Document.OpenMarkdown` builds a WordprocessingML package in memory (the pattern of the RTF and 2003 XML readers) and wraps it in the hub. The document reports `Format == OfficeFormat.Markdown` and saves as DOCX unless asked otherwise; saved unedited as Markdown it writes its source bytes. | Markdown | Document | |---|---| | ATX and setext headings | styles `Heading1` to `Heading6` (outline levels 0 to 5), with a bookmark `md_` so `[x](#slug)` links land on them | | paragraph | paragraph; a soft break is a space, a hard break is `w:br` | | emphasis, strong, strike-through | `w:i`, `w:b`, `w:strike` on the runs | | code span | character style `InlineCode` (Consolas) | | fenced and indented code | one paragraph per line in style `SourceCode` (the info string is dropped: `loss.markdown-code-language`) | | block quote | style `Quote`, indented by 720 twips per level | | bullet and ordered lists | numbering definitions (nine levels each; bullets `•` `◦` `▪`; decimal, lower letter, lower roman); every list is its own `w:num`, an ordered list carries its start number in a `w:startOverride`; further blocks of an item are indented paragraphs; tight lists use the list style's contextual spacing, loose ones switch it off | | task list items | a leading ☐ or ☒ in the item text | | thematic break | an empty paragraph with a bottom border | | tables | `w:tbl` in style `TableGrid`, first row marked as header and bold, column alignment as paragraph justification (left is the default) | | links, autolinks | `w:hyperlink`: external ones through a relationship, `#slug` ones as an anchor; targets other than web, mail, `tel:` and relative paths are kept as text (`loss.markdown-link-target`) | | images | embedded when the address is a `data:` URI or the caller's `MarkdownImportOptions.ImageResolver` returns bytes (PNG, JPEG, GIF, BMP); otherwise the description is kept as text and reported (`loss.markdown-image`, `loss.markdown-image-format`, `loss.markdown-image-size`). Nothing is read from a path or fetched from a URL by default. | | footnotes | `footnotes.xml` with the separator notes, `FootnoteReference` and `FootnoteText` styles | | front matter | a first line `---`, `key: value` lines and a closing `---` or `...`: title, author, subject, description and keywords go to the core properties; other lines are `loss.markdown-front-matter`. Text between two thematic breaks that is not YAML-shaped is not front matter. | | raw HTML (block and inline) | read by the HTML importer's fragment reader (sanitised, nothing fetched; see `html.md`) and written with the document's own styles: a block gives headings, paragraphs, lists, quotes and code; inline tags are matched with their closing tag among the siblings (`A bold word`) and give runs (bold, italic, underline, strike, code, links); `
` is a line break; comments are ignored. What the document model does not carry from it (scripts, styles, forms, table structure, pictures) is dropped and reported as `loss.markdown-raw-html`. | Resource use: input over `MaxPartSize` raises `OfficeLimitException`; pictures count against `MaxAllocationBytes`; invalid UTF-8 is replaced and reported (`markdown.encoding-replaced`). A UTF-8 or UTF-16 byte-order mark is read. Any other failure is a typed `OfficeFormatException` (`markdown.damaged`). The template engine's `markdown` tag uses the same parser (`MarkdownRich`): a Markdown value in a template and a Markdown file read the same way, then pass through the template vocabulary (which drops tables, pictures, rules and raw HTML, naming them in the `template.markup-dropped` warning). ## Export: the hub to Markdown `Document.ToMarkdown(MarkdownExportOptions?)`, `WriteMarkdown`, `SaveAs("x.md")`, `Office.Convert("a.docx", "a.md")`. `Document.ExportLoss` lists what Markdown could not say. - **Front matter** from the core properties (title, author, subject, description, keywords) when any is set. - **Headings** from a heading style, `Title` and `Subtitle`, or an outline level of 1 to 6 (resolved through the style chain). Bold that comes from the paragraph style is not emphasis; only direct formatting and character styles are. - **Lists** from numbering (`Numbering.LabelBody`): bullets or numbers, nesting by level, a different list (another numbering instance) gets the other marker (`-`/`*`, `.`/`)`) so adjacent lists stay apart. A paragraph with the list style or an indent after a list item is a continuation of the item, written indented under it; so is code. Empty items keep their marker. A ☐ or ☒ prefix becomes `[ ]` or `[x]`. - **Quotes** from the styles `Quote`, `Intense Quote`, `Block Text`; depth from the left indent in 720 twip steps. - **Code** from paragraphs in a style whose name says code (`Source Code`, `HTML Preformatted`, `Verbatim`…): consecutive lines become one fenced block, the fence longer than any backtick run inside. Runs in a monospaced font or a character style named code are code spans. - **Tables**: the first row is the header; alignment from the first body row's paragraphs; pipes are escaped, paragraphs of a cell are joined with `
`; merged cells become empty cells (`loss.markdown-merged-cells`); a table in a cell is flattened (`loss.markdown-nested-table`). - **Links** from `w:hyperlink` (relationship or anchor), simple `HYPERLINK` fields and complex field runs; an anchor on a heading becomes `#slug` with GitHub's slug and numbering of repeated headings. Anchors elsewhere are `loss.markdown-link-anchor`. - **Pictures**: `![alt](address)`. With `MarkdownExportOptions.ImageSink` the sink receives the bytes and returns the address; without one the address is `media/imageN.ext` and the bytes are not written (`loss.markdown-image-bytes`). - **Notes**: footnotes and endnotes become `[^n]` references and definitions at the end, in order of reference (endnotes: `loss.markdown-endnote`). - **Escaping**: `\ ` * [ ] < > ~` always, `_` at word edges, `|` in tables, `&` where it would start a character reference, and the starts of lines that would read as headings, list markers, setext underlines or ordered list numbers. - **Left out and reported**: underline, colour, highlight, super- and subscript and fonts (`loss.markdown-formatting`), equations (written as linear text, `loss.markdown-equation`), shapes, text boxes, charts and diagrams (`loss.markdown-drawing`), comments (`loss.markdown-comment`), headers, footers, page and section settings (not reported: layout is not content), tracked changes (insertions kept, deletions dropped). Round trip: every one of the 652 CommonMark examples goes through Markdown → document → Markdown → document; 642 give the same document on text, styles, list labels, table cells and link targets. The ten that differ are listed in `tests`: tabs and trailing white space inside text, empty headings (dropped), a list item whose only content is a quote or a heading, and one reference-style image. ## Workbooks and presentations `Workbook.ToMarkdown()` writes, per worksheet, a second-level heading with the sheet name and a pipe table of the used range. Cells appear the way the number format displays them (the same text as CSV with `Formatted`); formulas write their cached results; the first non-empty row is the header; columns whose body cells are all numbers are right-aligned; rows with no cell are left out, since a blank line would end the table. Losses: formulas, cell styles, merged cells, hidden sheets, chart and macro sheets, drawings and pivot tables (`Workbook.ExportLoss`). `Presentation.ToMarkdown()` writes a second-level heading per slide (`Slide 3: Title`), the slide's text in reading order (body placeholders as bullet lists by outline level, other text as paragraphs, tables as pipe tables), and speaker notes under `### Notes`. Bold and italic runs keep their emphasis. Pictures, shapes without text, charts, links, layout and animations are reported in `Presentation.ExportLoss`. ## Defects `markdown.too-deep` (Limit, repaired), `markdown.encoding-replaced` (Encoding, repaired), `markdown.damaged` and `markdown.export-failed` (Structure, fatal). See `defects.md`. Loss codes: `loss.markdown-*` for documents; `loss.formulas`, `loss.cell-styles`, `loss.merged-cells`, `loss.hidden-sheets`, `loss.sheet-kinds`, `loss.embedded-objects` for workbooks; `loss.pictures`, `loss.shapes`, `loss.links`, `loss.hidden-slides`, `loss.layout` for presentations. # OpenDocument (ODF 1.3; 1.2 and the 1.4 drafts read) OpenDocument files have a **native layer in the core package** (`RecursiveAI.OneDocument.Odf`) that the text, spreadsheet and presentation units share, and a **conversion into and out of the word-processing hub** in `.Documents`. This note covers both; spreadsheets and presentations add their own conversions on the same package layer. ## The package layer (core) | Type | What it is | |---|---| | `OdfPackage` | One ODF file: a zip package (OpenDocument 1.3 Part 3) or a flat XML document (`office:document`, Part 1), told apart by content. Entries as bytes, the four XML parts as lossless markup trees, the manifest, the style families and the metadata. | | `OdfManifest`, `OdfManifestEntry` | `META-INF/manifest.xml`: full path, media type, version, encryption flag. | | `OdfStyleSet`, `OdfStyle` | Common and automatic styles of `styles.xml` and `content.xml`: lookup by family and name, inheritance along `style:parent-style-name` ending in the family's `style:default-style`, `Resolve` of one property. Families: the `style:family` values plus `list`, `outline`, `page-layout`, `master-page`, `data`. | | `OdfMetadata` | `meta.xml`: title, subject, description, creator, last-modified-by, dates, language, generator, keywords, user-defined properties, document statistics. | | `OdfNamespaces`, `OdfMarkup`, `OdfMediaTypes` | Namespace names, child/attribute helpers, media types and the body element of each. | ```csharp using var pkg = OdfPackage.Open(ByteSource.FromFile("a.odt")); MarkupNode body = pkg.Body; // lossless markup tree of content.xml OdfStyle? s = pkg.Styles.Find("paragraph", "Heading_20_1"); string? size = pkg.Styles.Resolve(s!, "text-properties", OdfNamespaces.Fo, "font-size"); pkg.SetEntry("Pictures/a.png", bytes, "image/png"); // added to the manifest pkg.WriteTo(stream); // zip; pkg.WriteFlatTo(stream) for a flat file ``` **Round trip.** A package written without edits copies each entry raw (the stored bytes, CRC, method and time, never recompressed). A part whose markup tree was edited is written from the tree, which keeps unknown elements, attributes, prefixes, comments and order. The manifest is written again only when an entry was added, removed or retyped. A written package starts with an uncompressed `mimetype` entry with no extra field, as Part 3 asks, then the other entries in their order. A flat file writes its own tree. `WriteFlatTo` on a zip package merges the four parts under one `office:document` (namespace declarations united, font faces and automatic styles de-duplicated) and inlines packaged pictures as base64 `office:binary-data`. Flat → zip at the package level is not implemented; the hub converts through the model. **Tolerant reading.** Missing or misplaced `mimetype`, a missing or damaged manifest, manifest entries without a file and damaged zip directories are repaired and reported (`defects.md`). A part that is not well-formed XML, or whose root is wrong, raises its typed error when first used. Entry names that are empty, absolute or contain `..` are dropped. Encrypted packages (any `manifest:encryption-data`) raise `OfficeUnsupportedException`; ODF encryption and signatures are not implemented. The limits object passes through: part count, part size, expansion ratio, XML depth, no DTDs. ## Presentations (`Presentation`) ODP, OTP and FODP convert into the presentation hub and out of it the same way as text: see `odp.md`. ## Text documents (`Document`) `Document.Open` accepts ODT, OTT and FODT by content (the `mimetype` entry or `office:mimetype`), never the extension. The file is converted to an in-memory DOCX package, and the hub wraps it like any DOCX; `Document.Format` stays `Odt`, `Ott` or `Fodt`, and `Document.ImportLoss` lists what did not map. `SaveAs("x.odt" | "x.ott" | "x.fodt")` (or with an explicit `OfficeFormat`) writes OpenDocument from the hub, and `Document.ExportLoss` lists what it could not carry. `Office.Open`, `Office.ExtractText` and `Office.Convert` reach the formats through the registry. Direct entry points: `OpenDocumentText.ToPackage` for the DOCX package. **Untouched files.** An ODT/OTT/FODT read and saved in its own format without an edit writes the bytes it was read from. Any edit reaches the exporter, which writes a valid package of its own from the hub; a save to another OpenDocument format goes through the exporter too. ### Reading (ODF → hub) | ODF | Becomes | |---|---| | `style:style` (paragraph, text), `style:default-style` | `w:style` with `basedOn`, `next`, outline level; the default paragraph style becomes `w:docDefaults`; `Standard` becomes `Normal`, `Heading N`, `Title`, `Header`, `Footer`, `Footnote`, `Internet link` and the like take Word's built-in ids and names. Display names are used, `_20_`-style name encodings decoded | | automatic styles | direct formatting on the paragraph or run, with the nearest common ancestor as `w:pStyle`/`w:rStyle` | | `fo:`/`style:` text properties | bold, italic, underline styles, strikes, small caps, caps, colour, background (as shading), font size and fonts (latin, Asian, complex), super/subscript, spacing, scale, hidden text, relief, outline, shadow, language | | paragraph properties | alignment, indents, spacing, line height (percent, exact, at least), keep with next/together, break before page, widows/orphans, background, borders with padding, tab stops with leaders, writing mode | | `text:p`, `text:h` | `w:p`; headings carry `w:outlineLvl` | | white space | collapsed as Part 1 says (`text:s`, `text:tab`, `text:line-break` explicit) | | `text:list`, `text:list-item`, `text:list-header`, list styles, `text:outline-style` | `w:numPr` over `numbering.xml`; one abstract numbering per list style, one `w:num` per list that restarts; `text:continue-numbering`, `text:continue-list`, `text:start-value` map to reuse and `w:startOverride`; label alignment (or the older space-before/min-label-width) becomes indents | | `table:table` | `w:tbl` with a grid from the column styles (`style:rel-column-width` shares what the fixed columns leave), spans, covered cells (`w:vMerge`), header rows, row heights, nested tables, cell borders, background, padding and vertical alignment | | master pages and page layouts | one `w:sectPr` per run of paragraphs on a master page: size, orientation, margins (header and footer heights folded in), columns, page-number format; `style:header`, `-left`, `-first` become default, even and first headers, `evenAndOddHeaders` and `titlePg` | | `text:note` | footnotes and endnotes | | `text:a`, bookmarks, reference marks | `w:hyperlink` (external targets become external relationships and are never followed), `w:bookmarkStart/End` | | fields | `PAGE`, `NUMPAGES`, `DATE`, `TIME`, `TITLE`, `SUBJECT`, `AUTHOR`, `KEYWORDS`, `COMMENTS`, `FILENAME`, `REF`, `PAGEREF`, `NOTEREF`, `SEQ` as simple fields with the stored result; other fields keep their text | | `text:table-of-content`, `text:alphabetical-index` | a `TOC` / `INDEX` field around the stored entries | | `office:annotation`, `office:annotation-end` | comments with ranges | | `text:tracked-changes` | insertions and deletions (`w:ins`, `w:del` with `w:delText`) with author and date | | `draw:frame` with `draw:image` | inline or anchored DrawingML picture (position, wrap, alt text, title); embedded objects keep their preview picture | | `draw:frame` with `draw:text-box` | a VML text box with converted content | | `office:meta` | core, extended and custom properties | Reported as losses (`loss.*`): `unmapped-markup.`, `shapes` (lines, rectangles, custom shapes, groups), `embedded-objects` (formulas, charts and OLE objects keep only a preview picture), `forms`, `linked-images` (never fetched), `pictures` (missing or undecodable data), `picture-cropping`, `ruby` (the text above the base), `formatting-revisions`, `column-breaks`, `page-break-after`, `table-page-break`, `section-columns` (text sections with columns), `index-fields` (indexes other than contents and alphabetical), `page-background`, `formulas` (the MathML symbols are kept as text), `content-controls`. The text of a rectangle, custom shape or group is kept in a text box; the geometry is dropped (`shapes`). ### Writing (hub → ODF) The exporter reads the hub's markup tree and writes `content.xml`, `styles.xml` and `meta.xml` (`settings.xml` is not written). Styles map back by the same names; direct formatting becomes automatic styles (de-duplicated); lists become list styles (`WWNum`, one per `w:num`; a list that resumes after other content sets `text:continue-numbering`); tables keep spans, vertical merges, header rows, widths, borders (from the cell, the table and its table style), shading, margins and heights; each `w:sectPr` becomes a page layout and a master page with its headers and footers, and the first block of a later section names its master page; notes, comments with ranges, bookmarks, hyperlinks, fields (page number, page count, date, time, document properties, references, sequences, TOC), tracked insertions and deletions, pictures and text boxes (inline and anchored) are written. A page or column break inside a paragraph splits it, because ODF has no inline page break. Reported as losses: `content-controls` (read through), `equations` (written as linear text), `fields` (other fields keep their last result), `charts`, `smartart`, `shapes`, `pictures` (linked or missing data), `ole-objects`, `ruby`, `continuous-section-page-change`. ## Checks - Written ODT, OTT and FODT validate against the OASIS ODF 1.3 RelaxNG schema with Jing (`tools/validate/validate_odf.py`; ID/IDREF checking is switched off because Jing rejects the schema's own ID typing) and open in LibreOffice headless, which converts them to PDF and DOCX. - Text of the DOCX → ODT → hub round trip equals the source's; files LibreOffice writes from the same DOCX read into the hub with the same text (`testdata/odf`). - Converted DOCX validates against the ECMA-376 Transitional schemas. - Fuzz targets `odfpkg` (the package layer) and `odt` (the conversion in both directions). # Spreadsheets: the workbook hub and the cell store `RecursiveAI.OneDocument.Spreadsheets` holds the SpreadsheetML hub: `Workbook`, `Worksheet`, `Cell`, the cell store and the shared string table. This note states what they do; the XML documentation on each type gives the details. Defect codes are in `defects.md` (prefix `xlsx.`). A binary workbook (XLSB) is shown through the same model; `xlsb.md` says how, and what it keeps. ## What is read, when `Workbook.Open` (path, stream or `IByteSource`) identifies the content (never the extension), opens the package, and reads the content types, the package relationships, the workbook part and its relationships. Nothing else. | Part | Read when | |---|---| | the workbook part | open (it holds the sheet list) | | a worksheet part | the first use of that sheet's `Store`, a cell or its row, column or merge data | | the shared string table | a string cell is read as text, or a string is set | | styles, theme | when `Workbook.Styles`, a colour or `Cell.FormattedText` is first used | | tables | when `Worksheet.Tables` is first enumerated | | drawings, pivots, calc chain, … | never; they stay as stored | A chart sheet, dialog sheet or macro sheet is listed (`Worksheet.Kind`) and never opened. Opening never recalculates. ## The round-trip contract - A part nobody edited is copied by the package as stored (the raw zip entry), so a workbook opened and saved without edits has every part byte-identical. Reading a sheet does not edit it. - A sheet is **edited** when its store was changed (a value, style, formula, row property, a removed cell), when its columns or merged cells were set, or when `Worksheet.Rewrite()` forces it. An edited sheet is written from the store; everything else changes nothing else. - In an edited sheet every element around `sheetData` stays where it was, in order, with its prefixes, `mc:Ignorable`, `extLst`, `x14ac:` attributes and unknown elements: the bytes before and after `sheetData` are kept, and the few elements the model edits (`dimension`, `cols`, `mergeCells`) are changed in a small markup tree built from those bytes. Unknown attributes on rows, cells and formulas and unknown children of rows and cells are kept in side tables and written back unchanged. - What an edited sheet does **not** keep exactly: the `spans` hint on rows is dropped (it is derived; Excel rebuilds it); attribute order on a cell is `r`, `s`, `t`, then the unknown ones; a cell's children are written in the order `f`, `v`/`is`, unknown; numbers are written in the shortest form that reads back to the same double; a self-closed `` becomes an element with content when rows exist. Values, types, styles, formulas, row and column properties and merges read back equal (checked over the corpus; see the unit report). - A sheet that cannot be read (ill-formed XML, wrong root) raises a typed `OfficeFormatException` when it is first touched; it is never replaced by an empty sheet on save, because it stays an untouched part. ### What one cell edit changes An edit changes the part of its sheet, and the following, only when required: - **`xl/sharedStrings.xml`** when a string is set (new entries are appended; stored entries, rich text and phonetic runs are not rewritten; `uniqueCount` is exact, `count` is advisory). - **`xl/workbook.xml`** when a formula is set: `calcPr` gets `fullCalcOnLoad="1"` (created if absent; left alone when calculation is manual), so Excel recalculates on load. Nothing is calculated here. - **`xl/calcChain.xml`** (removed, with its relationship and content-type override) when a formula was removed or replaced, a formula cell was cleared, or the sheet list changed: Excel reports a damaged file for a chain that lists a cell without a formula or a sheet that is gone, and rebuilds the chain itself. Adding a formula needs no change to the chain. - `[Content_Types].xml` and `_rels` follow part additions and removals. ## The cell store `CellStore` (per sheet) keeps used cells only. Rows are sparse and sorted; each row keeps its cells as parallel arrays sorted by column ("struct of arrays"): a `ushort` column, a kind byte, an 8-byte payload, a 4-byte style index, and, in rows that hold formulas, a 4-byte index into the sheet's interned formula records. That is 15 bytes per cell (19 with formulas) plus about 250 bytes per row, with no object per cell. Lookup is two binary searches; iteration by row is a linear scan of dense arrays; iteration by column visits each row once (`EnumerateColumn`). | Kind byte | Meaning | |---|---| | bits 0-2 | `CellValueKind`: empty, number, text, boolean, error | | bit 3 | a formula is attached (the value is its cached result) | | bit 4 | the text is an ISO 8601 date (`t="d"`), written back as such | ### Allocation The store itself allocates nothing per cell: a row is five parallel arrays (columns, kinds, values, styles, formula indices), so a cell costs about 15 bytes (19 with a formula). `Cell` is a class and `sheet["B2"]` makes a transient handle for convenience; iteration (`EnumerateRows`, `EnumerateColumn`, `CellRowView.ValueAt`/`CopyValues`) uses only structs and allocates nothing, which a test asserts. Parsing a sheet allocates about one extra byte per live byte (rows' arrays grow by doubling), not per-cell objects. A saved sheet is built in memory before it replaces the part, so one sheet part larger than about 2 GB of XML cannot be written until the streaming writer exists. ### The value representation `CellValue` is a 16-byte value type: a kind, a double (numbers; 0 or 1 for booleans) and an int. Reading a cell allocates nothing. The calculation layer builds its array value model on it. - **Number**: the double. Dates stay numbers (the 1900/1904 rules are another layer's). - **Boolean**: 1.0 or 0.0. - **Error**: a `CellError` number (`#NULL!` … `#BUSY!`). An error text the table does not know reads as `CellError.Unknown` and keeps its text in the sheet pool. - **Text**: a handle, not a string. A non-negative handle is an index into the shared string table; a negative handle is the complement of an index into the sheet's own string pool (inline strings, text results of formulas, ISO dates). `Worksheet.GetText(value)` or `Cell.Text` resolves it. Handles are valid for the workbook they came from. - **Formula**: kept as text with its kind (`Normal`, `Shared`, `Array`, `DataTable`), its range and shared index, and the other attributes of `f` as written. The cells of a shared formula that are not its anchor have empty text and the shared index; translating the anchor's text for them is the formula layer's job. Identical records are interned, so a shared formula costs one record per `si`. Text goes through ST_Xstring: `_x000D_` style escapes are decoded on read and written for characters XML cannot carry (and for text that looks like an escape). ### Row and column data Row properties are in the store (`Worksheet.GetRow(n)`: height, hidden, outline level, collapsed, style); column runs (`Worksheet.Columns`, `SetColumns`), merged ranges (`MergedCells`, `Merge`, `Unmerge`), the declared dimension and frozen panes are read from the markup around `sheetData`. Writing the column and merge lists replaces those elements at their schema position; the `dimension` of an edited sheet is set to the used range when the part had one. ## Sheets `Workbook.Sheets` lists the sheets in order (`Name`, `State`, `Kind`, `SheetId`). `Add`, `Insert`, `Remove`, `Move` and `Worksheet.Rename` edit the sheet list. They keep each existing `sheet` element exactly as written, remap `localSheetId` of defined names and the active tab, remove names scoped to a removed sheet (and an emptied `definedNames`), and remove a deleted sheet's part and the parts only it uses (drawings, tables, comments). **Renaming does not rewrite formulas, defined names or chart references that mention the old name; that belongs to the formula layer.** A workbook keeps at least one visible sheet. `Workbook.Create()` writes the same bytes on every run. ## Saving `Save()` is the package's atomic replace (temp file and rename). `SaveAs(path)` picks the format by extension (`.xml` is Flat OPC); `SaveAs(stream, format)` writes XLSX, XLSM, XLTX, XLTM, XLAM or Flat OPC. Changing between the zip formats changes the main part's content type only; saving a workbook that holds a macro project as XLSX or XLTX is refused (`OfficeUnsupportedException`). The workbook's own `Format` does not change. ## Measured Bytes per cell, load time and lazy open on the generated stress workbooks (1M rows by 20 columns, 133 MB, 20,000,020 cells with styles, strings and 4M formulas): 37.2 retained bytes per cell (ClosedXML 0.105.1: 78.8, NPOI 2.7.6: 360.4); a full load takes about 9 s and a forced rewrite of the sheet about 8 s on a shared 32-core x64 host (indicative; method and the other sizes in `bench/README.md`). Opening reads 0.5 MB of the file (the zip directory, `[Content_Types].xml`, relationships and `workbook.xml`); the first sheet is read when it is touched, and the string table is not read until a string cell is. Peak working set for the load and rewrite of the stress file is 2.6 GB, mostly the XML of the one saved sheet built in memory. ## Limits `MaxFormulaLength` bounds a formula (read and set); `MaxRecordCount` bounds the cells of a sheet and the entries of the string table; `MaxXmlDepth` and the other XML limits apply to every part; a row may not hold more than 2^20 cells before the duplicates are resolved. The sheet reader streams the part: memory is the store, not the XML. ## Known limits of this layer - Styles are indices on the cell store; the style sheet, number formats, dates, tables, conditional formatting, data validation and defined names are in `spreadsheets-styles.md`. - A UTF-16 sheet part is read and written as UTF-8 (the declaration is replaced). - Adding a string looks for an equal string only among those added in the session. # Spreadsheets: styles, tables, conditional formatting, validation and names `RecursiveAI.OneDocument.Spreadsheets` models the rest of the workbook around the cells (ECMA-376 Part 1 §18). Every model is a view over the part's markup tree: what the model does not read (extension lists, unknown attributes and elements, attribute order, prefixes) stays where it was when the part is saved, and a part nobody touched is copied as stored. Reading a feature never marks the workbook modified. Defect codes are in `defects.md` (`xlsx.`). ## Styles `Workbook.Styles` (`StyleSheet`) is read from the `styles` part the first time it is used. A workbook without one gets a default sheet in memory (Calibri 11, one cell format, `Normal`); it becomes a part only when something is added. | Collection | Read | Add | |---|---|---| | number formats | `NumberFormats` (id to code), `GetNumberFormatCode(id)`, `GetNumberFormat(formatIndex)` (parsed, cached) | `GetOrAddNumberFormat(code)`: a built-in code reuses its id, an existing custom code its id, else the next id from 164 | | fonts, fills (pattern and gradient), borders | `GetFont`, `GetFill`, `GetBorder` | through `GetOrAdd` | | cell formats (`cellXfs`) | `GetCellFormat(i)` as stored; `Resolve(i)` as effective | `GetOrAdd(CellStyleSpec)` | | named cell styles (`cellStyles` + `cellStyleXfs`) | `CellStyles`, `GetStyleFormat(i)` | `AddNamedStyle(name, spec)` | | differential formats | `GetDifferentialFormat(i)` | `AddDifferentialFormat` | | table styles | `TableStyles`, `DefaultTableStyle` | read only | ### Resolution `Resolve(formatIndex)` returns a `ResolvedCellStyle` (font, fill, border, alignment, protection, number format id and code, quote prefix, the named style's name), one object per format, so cells cost nothing. Rules: - A cell format's own `fontId`, `fillId`, `borderId` and `numFmtId` are always used; the `applyX` flags are not consulted. Checked in Excel 16.0.20527 with generated cell formats whose named style has 0% and whose own format is General or 0.00%, with `applyNumberFormat` false, absent and true: Excel shows the cell format's own format in all cases. An alignment or protection child missing from the cell format is taken from its named style. - An index past the last cell format resolves like format 0. A missing id (font 9 of 3) resolves to the default font, an empty fill or an empty border. - A `numFmt` in the file overrides a built-in id of the same number; otherwise built-in ids come from the US-English table in `NumberFormat`. - Colours: `ResolveColor` gives ARGB. Indexed colours use the file's `indexedColors` when present, else the default palette of §18.8.27 (64 is the system foreground, 65 the background). Theme colours use the workbook's theme part (`lt1, dk1, lt2, dk2, accent1–6, hlink, folHlink` for indexes 0–11); a workbook without a usable theme (defect `xlsx.theme-unreadable` when it is broken) uses the current Office colours. A tint is applied in HLS space as §18.3.1.15 describes (lighter: `L' = L(1-t)+t`, darker: `L' = L(1+t)`); `8FAADC` for accent 1 at +0.4 matches what Excel names "Lighter 40%". - A solid fill in a differential format shows its **background** colour (Excel's convention); `CellFill.Foreground`/`Background` are the stored attributes. ### Interning `GetOrAdd(spec)` reuses an existing cell format that has the same number format, font, fill, border, alignment, protection, quote prefix and named style, and otherwise appends one. Fonts, fills and borders are interned by their canonical XML. Apply flags are not compared; a new format sets `applyX="1"` for each property that differs from its named style's. A format with attributes or children the model does not read (`pivotButton`, `extLst`) is never reused. Appending changes `count` of the collection and leaves everything else in the part alone. A cell's `StyleIndex` stays a plain integer: cells hold no style objects. ### Formatted text and dates `Cell.FormattedText` formats a numeric cell through its resolved number format with the workbook's date system (`workbookPr/@date1904`, `Workbook.DateSystem`), text through the format's text section, booleans as TRUE/FALSE. General uses the width of a default column (11 characters); `GetFormattedText(width)` takes another. `Cell.TryFormatNumber` writes into a span without allocating. `Cell.IsDate`, `Cell.DateValue` and `Cell.SetDate` use the same rules; `SetDate` gives a cell whose format does not show dates the built-in date format. The format engine is `RecursiveAI.OneDocument.NumberFormats` (see `number-formats.md`). ## Tables `Worksheet.Tables` reads the sheet's `tableParts` and their parts when first enumerated. `Table` exposes id, name, display name, range, header and totals row counts, data range, autofilter range, columns (id, name, totals function and label, calculated column formula) and the style name. Structured-reference evaluation belongs to the calculation layer. - `Add(name, range, hasHeaderRow, style)` writes the header cells (a text cell keeps its text, a repeated name gets a number appended, anything else becomes `ColumnN`), creates `/xl/tables/tableN.xml` with the next free table id, the sheet relationship and the `tablePart` element (declaring `xmlns:r` on the sheet root when it is not in scope). - `Resize(range)` changes `ref`, the autofilter (without the totals row), adds or removes columns at the end (new names `ColumnN`, unique) and rewrites header cells that do not hold their column's name. It refuses a range that overlaps another table or puts a merged cell in the header. - `Rename` and `SetStyle` change the attributes. `Remove` drops the part, its relationship and the element; cells keep their values. - A table name must be valid as a defined name, must not look like a cell reference, and must be unique among the workbook's tables and workbook-scoped names (case-insensitive). - Table parts are rewritten only when edited. Removing a sheet already removes the tables only it uses. ## Conditional formatting `Worksheet.ConditionalFormats` lists the `conditionalFormatting` groups of the sheet body and then the x14 groups in `extLst` (`IsExtended`). `ConditionalFormatRule` reads type, priority, `dxfId`, `stopIfTrue`, operator, text, time period, rank/percent/bottom, above-average, the formulas (text), colour scales, data bars and icon sets (value points included). `Add(ranges, spec)` builds a standard rule (`ConditionalRuleSpec.Expression`, `CellIs`, `TwoColorScale`, `ThreeColorScale`, `DataBars`, `Icons`, `Top`, `Duplicates`) in the schema position after `mergeCells`/`phoneticPr` and before `dataValidations`, with the next free priority across both forms (so it is evaluated last; set `Priority` to reorder). `Remove` deletes a group, and an emptied x14 container with it. Priorities and `dxfId` can be edited in place. Extended rules are not authored. ## Data validation `Worksheet.Validations` lists the `dataValidation` elements and the x14 ones (which hold lists and formulas that refer to other sheets in older files). A validation reads type, error style, operator, allow-blank, the in-cell drop-down (the file stores the inverse `showDropDown`), messages, ranges and both formulas as text. `Add(ranges, spec)` takes `DataValidationSpec` (`List`, `ListFromRange`, `WholeBetween`, `DecimalBetween`, `Custom`, or the properties) and checks Excel's limits (titles 32 characters, messages 255, an inline list 255). `Remove` deletes one and drops an emptied container. An existing validation is replaced by removing it and adding a new one. ## Defined names `Workbook.Names` lists the `definedName` elements in file order: name, scope sheet, formula text, hidden, comment, function/macro flags, and the built-in kind for `_xlnm.` names. `Add(name, formula, scope, hidden)`, `AddBuiltIn(kind, sheet, formula)`, `Rename`, `Remove` and `Find(name, scope)` (case-insensitive) edit only `definedNames`; an emptied container is removed because the schema needs one name inside it. `IsValidName` applies Excel's rules (first character a letter, underscore or backslash; letters, digits, `_ . ? \` after; at most 255 characters; not a cell reference in A1 or R1C1 form). Renaming a name or a sheet does not rewrite formulas that use the old text, and formula text is not parsed: that belongs to the formula layer. Sheet moves keep `localSheetId` right, as before. ## Limits - Counts are bounded by `MaxRecordCount` through the markup tree; every part counts against `MaxPartSize` and the XML depth limits. - Resolution allocates one object per cell format, on first use. Reading cells does not touch the style sheet. - A damaged table part is ignored (`xlsx.table-unreadable`) and kept as stored. A styles part that is not a style sheet raises `xlsx.styles-wrong-root` when first used, and cells still read. # Spreadsheets: streaming cursors `RecursiveAI.OneDocument.Spreadsheets.Streaming` reads and writes XLSX, XLSM and Flat OPC workbooks as a forward-only stream of rows and cells, beside the workbook hub (`spreadsheets.md`), not through it. Use it when the sheet is larger than you want in memory or when you only need to pass the values through once. Both sides use the hub's `CellValue`, `CellValueKind`, `CellError`, `CellAddress` and `ColumnProperties`, so a value means the same thing in both. Defect codes are in `defects.md` (prefix `xlsx.`). ## Reading ```csharp using XlsxStreamReader book = XlsxStreamReader.Open("big.xlsx"); using SheetCursor sheet = book.OpenSheet("Data"); while (sheet.ReadRow()) { while (sheet.ReadCell()) { switch (sheet.Kind) { case CellValueKind.Number: total += sheet.Number; break; case CellValueKind.Text: Consume(sheet.TextUtf8); break; // span, no allocation } } } ``` - **Opening** reads the package directory, content types, relationships and the workbook part. The shared string table and the style sheet are read the first time a cell asks for text or a date. - **Row** (valid after `ReadRow` returns true): `RowNumber`, `RowHeight` (-1 when none), `IsRowHidden`, `HasCustomRowHeight`, `HasCustomRowFormat`, `IsRowCollapsed`, `RowOutlineLevel`, `RowStyle`. `ReadRow` skips the cells of the previous row that were not read. - **Cell** (valid after `ReadCell` returns true, until the cursor moves): `Column`, `Address`, `Style`, `Kind`, `Value` (a `CellValue`), `Number`, `Boolean`, `Error`, `SharedStringIndex`, `TextUtf8`, `TryCopyText`, `GetText`, `IsIsoDateText`, and for formulas `HasFormula`, `FormulaKind`, `FormulaUtf8`, `GetFormula`, `FormulaRangeUtf8`, `SharedFormulaIndex`. `IsDate` and `TryGetDateTime` go through the style's number format and the workbook's date system. - **Cached values** are what the cursor returns; opening never recalculates. A formula cell reports the kind of its cached result. - **Text** is a span of UTF-8 with references resolved and rich-text runs joined (phonetic runs left out). `_xHHHH_` escapes are resolved by `TryCopyText` and `GetText`, not by the span. - **Order.** Rows and cells are delivered in the order the part stores them. A part that stores them out of order is reported once per row (`xlsx.stream-row-order`, `xlsx.stream-cell-order`) and delivered as stored; the hub sorts and merges instead. Cursor and hub give equal values for every in-order sheet. - **Memory.** Bounded by the largest token of the sheet part (a long string or formula), plus the shared string table as plain text (the text of every entry plus four bytes per entry) when text is read. Cell data is never held. - **Allocation.** Zero per cell: numbers, booleans, errors, styles, indexes and spans allocate nothing; only `GetText`, `GetFormula` and the other string-returning members do. A shared string is built once and reused for every cell that uses it, up to a budget of about 8 million characters (about 16 MB); past that, strings are built per call. - **Failures.** Ill-formed XML or a wrong root raises `OfficeFormatException`; limits raise `OfficeLimitException`; both carry the part and byte offset. Repairs (`xlsx.cell-address-invalid`, `xlsx.cell-value-invalid`, `xlsx.row-index-invalid`, …) go to `OpenContext.Defects` with the cell address. The cancellation token given to `Open` is checked on every `ReadRow`. ### The sheet scanner A worksheet part is machine-written UTF-8 and is almost all of the work, so the cursor reads it with a scanner built for that shape instead of the general tokenizer: vectorised tag and quote search, attributes sliced on demand, a text-plus-end-tag shortcut for ``, and an exact shortcut for plain decimals of up to 15 digits (a whole number below 2^53 divided by a power of ten is one correctly rounded division, equal to the full parser's result). It reports the same tokens as the general tokenizer for every well-formed part, and hands the part to the general tokenizer for UTF-16. It is more tolerant than the tokenizer in three ways: end tags are matched by depth, not by name; a malformed attribute list ends early; namespace declarations are not checked. It still refuses DTDs, truncated parts, stray end tags and the depth, name and allocation limits. The `stream` fuzz target runs every sheet through both and requires equal cells wherever the tokenizer accepts the part. ## Binary workbooks: `.xls` and `.xlsb` `XlsStreamReader` (legacy Excel workbooks, BIFF8 and the older BIFF2 to BIFF5 layouts) and `XlsbStreamReader` (the binary OOXML workbook) have the same shape as `XlsxStreamReader`: `Open(path | stream | byte source)`, `Sheets`, `DateSystem`, `OpenSheet(index | name)`, `SharedStringCount`, `GetSharedString`, `IsDateStyle`, `GetNumberFormatId`. Their cursors, `XlsSheetCursor` and `XlsbSheetCursor`, have the cursor members above that a binary workbook can answer (`ReadRow`, `ReadCell`, `RowNumber`, `Column`, `Address`, `Style`, `Kind`, `Value`, `Number`, `Boolean`, `Error`, `SharedStringIndex`, `TextUtf8`, `TryCopyText`, `GetText`, `HasFormula`, `FormulaKind`, `IsDate`, `TryGetDateTime`, the row properties and `Columns`/`DeclaredDimension`). `Spreadsheet.Read`, `Spreadsheet.ReadRows` and `Office.ExtractText` read all three formats through them; the workbook model is never built. - **Memory.** A cursor reads its sheet's records from the file through a 128 KiB buffer and holds one record. The shared string table is held as one pool of UTF-16 text; strings built by `GetText` are kept up to about 8 million characters. A password-protected `.xls` is decrypted whole into memory first, as the model path does. - **Allocation.** Zero per cell for numbers, booleans, errors, styles, indexes, spans and `TryCopyText`. A text cell that is not a shared string and is split by a `CONTINUE` record, or carries rich-text runs, allocates for that cell. `GetText` allocates the first time a shared string is asked for. - **Rows.** A BIFF sheet stores a `ROW` record (height, outline, hidden, format) before the cells of the row. The cursor delivers the rows that have cell records, with the properties of their `ROW` record; a row without cells is not delivered. Cells that hold only a format come as `CellValueKind.Empty` with their style. - **Order.** Rows and cells come in the order the file stores them; out-of-order input is reported once per row (`xls.stream-row-order`, `xls.stream-cell-order`) and delivered as stored. Cursor and model give equal values for every in-order sheet (`BinaryCursorTests` compares both over every `.xls` and `.xlsb` fixture, cell for cell). - **`.xls` details.** `Style` is the XF index; for BIFF2, which has none, a number per distinct attribute combination, in the order the model numbers them. Formula cells report the cached result, `HasFormula` and `FormulaKind` (from the first byte of the formula tokens and the shared-formula flag); the tokens are not decoded, so there is no formula text. Chart, dialog and macro sheets are listed in `Sheets` and cannot be opened. A BIFF2 to BIFF4 file keeps its formats in the sheet records, so opening one reads the whole file once. - **Failures.** A stream that ends inside a record ends the sheet at the last whole record and reports `biff.truncated-record` (`.xlsb`: `xlsb.record-truncated`) with the record offset; a stream that does not start with a `BOF` record raises `OfficeFormatException` (`biff.no-bof`); the record-count and allocation limits raise `OfficeLimitException`. Damage that is none of these surfaces as `biff.malformed`, never as an index or cast exception. ## Writing ```csharp using XlsxStreamWriter book = XlsxStreamWriter.Create("out.xlsx"); int money = book.AddFormat(new StreamCellFormat { NumberFormat = "#,##0.00" }); SheetStreamWriter sheet = book.AddSheet("Data", new SheetWriteOptions { FreezeRows = 1 }); for (int r = 1; r <= rows; r++) { sheet.BeginRow(); sheet.WriteText(name[r]); sheet.WriteNumber(amount[r], money); } sheet.Complete(); book.Finish(); ``` - Rows are written in order; `BeginRow` starts the next (or a given) row, cells go left to right (`MoveToColumn`, `SkipColumns` leave gaps). `WriteNumber`, `WriteText`, `WriteBoolean`, `WriteError`, `WriteBlank`, `WriteDate`, `WriteValue`; formulas as text with an optional cached result (`WriteFormula`, `WriteFormulaNumber/Boolean/String/Error`). Formula text and style indexes are checked before the cell starts, so a bad argument leaves the sheet writable. - Styles come from `AddFormat(StreamCellFormat)` (font, fill, border, alignment, number format) or from a caller-supplied `XlsxWriteOptions.StylesXml`. Column widths and hidden/outline state, freeze panes, gridlines and sheet state are `SheetWriteOptions`; `Merge` records merged ranges. - Strings are shared (up to `MaxSharedStrings` distinct strings, then inline) or all inline, per `XlsxWriteOptions.StringStorage`. - The writer has its own streaming zip (data descriptors; `Zip64` for parts or archives past 4 GB). Memory is constant in the number of rows: the working set is the shared strings table (capped) plus a few buffers. - Output validates against the pinned ECMA-376 transitional and OPC schemas and is deterministic: no clock or random value is written. ## Measured (indicative; see the unit report for machines and loads) Read, 1M x 20 stress sheet: 0.002 bytes allocated per cell, peak working set unchanged from 10k to 1M rows. Write, 1M x 20: constant working set of about 39 MB, 0.03 bytes per cell. # Spreadsheets: rows of plain objects `Spreadsheet.Read` and `Spreadsheet.Write` move rows of plain classes, structs and records to and from XLSX, CSV and TSV in one call. They sit on the streaming cursors (`spreadsheets-streaming.md`) and the CSV layer (`csv.md`), so memory stays constant however many rows there are. Nothing uses reflection: a source generator, run when you compile, writes the reading and writing code for each type, so the binding works under trimming and NativeAOT. Defect codes (`binding.`) are in `defects.md`. ```csharp [SpreadsheetRow(Sheet = "Orders")] public partial record Order(int Id, string Customer, decimal Total, DateOnly Placed, Status? State); Spreadsheet.Write("orders.xlsx", orders); // or .csv, .tsv foreach (Order o in Spreadsheet.Read("orders.xlsx")) { } // lazy, streaming await foreach (Order o in Spreadsheet.ReadAsync(stream)) { } IEnumerable raw = Spreadsheet.ReadRows("orders.xlsx"); // no type: one array per row ``` ## Marking a type `[SpreadsheetRow]` goes on a `partial` class, struct, record or record struct (and on nothing generic). The generator adds the `ISpreadsheetRow` implementation that `Read` and `Write` require as their constraint, so a type that is not marked does not compile into the call. Nested types work when every containing type is `partial` too. Type options (`SpreadsheetRowAttribute`): | Option | Default | Meaning | |---|---|---| | `Sheet` | first visible worksheet / `Sheet1` | The sheet to read and to write. Ignored for CSV and TSV. | | `Strict` | false | Lenient reading reports a bad cell or a missing column as a defect and keeps the default; strict reading throws at the first one. | | `HasHeader` | true | The first row holds column names. Without it, columns are found by position or `Index`, and writing leaves the header out. | Strictness is a property of the type, not of the call: the statics take no options object. ## Which members are columns - Public instance properties and fields, in declaration order. Private members only when marked `[SpreadsheetColumn]`. - A get-only property that nothing fills (no constructor parameter of that name) is taken for a computed value and left out. Marking it `[SpreadsheetColumn]` writes it and never reads it. - A set-only property is read and not written. - Reading uses the parameterless constructor when the author wrote one; otherwise the constructor with the most parameters (a record's primary constructor), each parameter filled from the column of the same name (ignoring case); then an object initializer sets the remaining writable members, `init` and `required` ones included. A constructor parameter with no member is error OOBIND003; one whose member is ignored gets `default`. `[SpreadsheetColumn]` options: a name (the header text; default the member name), `Order` (marked columns first, ascending, then the rest in declaration order), `Index` (a fixed 1-based sheet column, used for reading and writing instead of the header), `Ignore`, `Format` (an XLSX number format code), `EnumAsNumber`. ## Supported types `string`, `bool`, `byte`, `sbyte`, `short`, `ushort`, `int`, `uint`, `long`, `ulong`, `float`, `double`, `decimal`, `DateTime`, `DateOnly`, `TimeOnly`, `DateTimeOffset`, `Guid`, any enum, and `Nullable<>` of the value types. Anything else is warning OOBIND001 and the member is left out. ## Reading - **Header.** The first row with a cell is the header. A header cell matches a column by name, ignoring case and surrounding white space. Extra sheet columns are ignored. A column the sheet lacks is `binding.column-missing` (reported once, member keeps its default; strict types fail only for members that cannot be null). A repeated header uses the first and reports `binding.header-duplicate`. - **Rows.** Rows in which no mapped column holds a value are skipped (blank lines, formatted-but-empty rows). The value of a formula cell is its cached result. - **Empty cells.** An empty cell leaves the default (`null` for nullable members, `""` for a non-nullable string) silently; strict types raise `binding.cell-empty` for a member that cannot be null. - **Coercions** follow Excel's: text that looks like a number reads as a number; a number reads as TRUE unless it is 0; TRUE reads as 1; a number reads as text the way it is stored (a date-formatted number as ISO 8601). An integer member takes a number only when it is whole and in range, otherwise `binding.cell-unconvertible`. A `decimal` from a number cell has the 15 significant digits Excel keeps; from text it keeps all of them. - **Dates and times.** A number is a serial of the workbook's date system (1900, with its leap-year quirk, or 1904); ISO 8601 and other invariant-culture date text also parse. `TimeOnly` takes the fraction of the day, rounded to the millisecond. A `DateTimeOffset` has offset zero (a spreadsheet keeps no time zone), or the offset of the text it came from. - **Enums** read from a name (any case) or a whole number that is a defined value. - **Format by content.** XLSX, XLSM, templates and Flat OPC go through the streaming cursor; CSV and TSV through the CSV reader (delimiter detected, TSV forced to tab). A file with only a header line cannot be recognised by content, so `.csv`, `.tsv` and `.txt` extensions are the fallback. - **Allocation.** The row buffer is reused: reading allocates the objects and the strings of string members (a string shared by many cells is built once by the cursor), nothing per column. ## Writing - The header row is bold, the first row is frozen, and columns are wide enough for their header and content. Dates are written as serials with a date format (`yyyy-mm-dd` for `DateOnly`, `yyyy-mm-dd hh:mm:ss` for `DateTime` and `DateTimeOffset`, `hh:mm:ss` for `TimeOnly`), so Excel shows and sorts them as dates. - `DateTime.MinValue` (an unset date) is written as an empty cell; any other date before the date system's start raises `binding.value-out-of-range`. The kind and offset of a date are dropped. - Whole numbers beyond ±2^53 cannot be held by a spreadsheet number, so they are written as text of their digits and read back exactly. NaN and infinity are written as an empty cell. A `decimal` is written as a double in XLSX (15 significant digits) and in full in CSV. - Enums are written by name (a value with no name as its number); `EnumAsNumber` writes numbers. A `Guid` is text in the 36-character form. `bool` is a boolean cell (`TRUE`/`FALSE` in CSV). - CSV and TSV use the CSV writer's defaults, including the formula-injection escape: a text that starts with `=`, `+`, `-` or `@` is written with a leading apostrophe. Reading such a file back gives the apostrophe. - Writing to a path replaces the file atomically; cancellation leaves the old file. XLSX output is deterministic. - At most 1,048,575 rows below the header; more raises `OfficeLimitException`. ## Generator diagnostics | Id | Severity | Meaning | |---|---|---| | OOBIND001 | warning | A member's type cannot be a column; it is left out. Mark it `[SpreadsheetColumn(Ignore = true)]` to silence. | | OOBIND002 | error | The type or a containing type is not `partial`. | | OOBIND003 | error | A constructor parameter matches no property or field. | | OOBIND004 | warning | Two columns have the same name (ignoring case). | | OOBIND005 | error | The type, or a containing type, is generic. | | OOBIND006 | warning | `Index` is outside 1 to 16384 and is ignored. | | OOBIND007 | warning | The type has no columns. | ## Not covered Types you cannot edit (from another assembly) cannot be marked; wrap them in a row type. There are no custom converters yet, and no column-by-name lookups on the non-generic `ReadRows` (it returns values positionally, with dates as `DateTime`). # Workbook calculation Namespace `RecursiveAI.OneDocument.Spreadsheets.Calc`, in the `.Spreadsheets` package, over the evaluator in `calc.md`. Entry points: `Workbook.Recalculate`, `Workbook.RecalculateAll`, `Cell.SetDynamicArrayFormula`, and `onedocument recalc`. Excel is the specification. Statements below marked *(pending the Excel check)* are first readings: the workbooks that settle them are generated by `tools/CalcSweep` (`gen`), have been run through the evaluator here, and are to be recalculated by Excel and compared with `CalcSweep compare` at the release gate; none of them has been compared with Excel yet. ## When it runs Opening never calculates. `Recalculate()` is the explicit command and calculates whatever the calculation mode says: a workbook set to manual stays manual (the `calcPr` is not touched) and the call is "calculate now". Edits made through `Cell` (values, formulas, clearing) are noticed; a plain `Cell.Formula` edit still sets `fullCalcOnLoad` as before and nothing is calculated until you ask. The first call builds the dependency graph and calculates everything. Later calls calculate the formulas an edit made dirty, plus the volatile and the run-time-reference ones. `RecalcOptions.Full` (or `RecalculateAll`) calculates everything. ## The graph One node per formula cell (shared formulas are translated from their anchor; data tables are left alone). The reverse index maps a cell to the formulas that read it: - a block of up to 16 cells is stored as one entry per cell; a bigger block as one entry per column (rows `first..last`), found by a binary search and a running maximum; a block wider than 256 columns sits in a short list. `SUM(A1:A100000)` is one entry. - defined names are expanded: a formula using a name depends on what the name's formula reads (a name holding a lambda reads nothing, its calls' arguments are walked). Sheet-scoped names hide workbook names on their sheet. - table references become areas at build time; `[#This Row]` is the cell on the formula's own row. - 3-D references depend on the cell on every sheet between the two (workbook order, either way round). - whole-row and whole-column references are blocks; when read they are clipped to the sheet's used extent (`ICalcResolver.ClipArea`), so `SUM(A:A)` reads the used rows only. `ROWS(A:A)` still answers 1,048,576 because it asks the reference, not the values; a function that inspects the *shape of the array* made from a whole column sees the used extent. - references to other workbooks add no edges: they are answered from the file's cache or a resolver (see "Other workbooks"). A formula that reads one is marked, so it is calculated again whenever a resolver is supplied. A formula edit, a sheet/name/table change or a different function registry rebuilds the graph on the next call (a pass over the formulas, no evaluation); a value edit costs only its dependents. After a rebuild only the edited cells' dependents are recalculated, unless a sheet, name or table changed, which recalculates everything. ## Levels and threads Formulas whose inputs are all finished form a level. A level runs on all the cores (`RecalcOptions.MaxDegreeOfParallelism`, default one per processor; levels smaller than `ParallelThreshold` run on the calling thread); results are written to the grid afterwards, sequentially, in cell order. Formulas on one level never read each other, so the answers do not depend on the thread count (tested at 1, 2 and 8). Volatile functions give different answers per call by nature. A formula that builds references at run time (a function marked `FunctionTraits.DynamicReference`: `INDIRECT`, `OFFSET`) cannot name its inputs in advance. Such formulas run alone, after the other formulas of their level, and when one reads a formula cell that has not been calculated yet, that cell is calculated first, inline (nesting limit 200). The dynamic ones and the volatile ones are dirty on every call. ## Circular references When no formula is ready but some are waiting, what is left is cycles and the formulas below them. The formulas on a cycle (strongly connected sets, and a formula that reads itself) get the value 0 and a `calc.circular-reference` defect; the formulas below them then calculate from those zeros. With iterative calculation on (below) the cycle is iterated instead. *(pending the Excel check: that Excel shows 0 for every cell on the cycle and calculates the others from it; workbook `circular`)* ## Iterative calculation `calcPr iterate` turns it on, `iterateCount` (default 100, at most 32767) and `iterateDelta` (default 0.001) bound it (ECMA-376 Part 1 §18.2.2). `Workbook.IsIterativeCalculation`, `IterationCount` and `IterationDelta` read them and `SetIterativeCalculation` writes them. Off, a cycle is cut to 0 as above. When nothing is ready but cycles remain, each cycle (a strongly connected set, or a formula reading itself) whose inputs from outside itself are finished is iterated, so a cycle that feeds another is settled first. A round calculates the cycle's formulas once each, in cell order (sheet, row, column), and each formula sees what the earlier ones of the same round wrote. The first round starts from the values the cells hold (the cached values; empty counts as 0). Iteration stops after a round in which no number moved by `iterateDelta` or more (Microsoft: values "change by less than" the maximum change; with a maximum change of 0 only a round with no change at all stops it), or after `iterateCount` rounds. A cell that changes between kinds (number to text, error to number) always counts as moved. A cycle that is still moving after the last round keeps its last values and reports `calc.iteration-limit`. `RecalcResult` counts `IterationRounds`, `IteratedCells` and `UnsettledCycles`. The formulas below a cycle are calculated after it settles. *(pending the Excel check: the round count and stopping rule, the order inside a round, and the values after a cut-off round; workbooks `iterate-self`, `iterate-count`, `iterate-pair-order`)* ## Data tables A `t="dataTable"` formula on the first cell of a block is a what-if table (ECMA-376 Part 1 §18.3.1.40): `ref` is the block of results, `dt2D` marks a two-variable table, `dtr` says a one-variable table has its input values in a row (else in a column), `r1` is the first input cell and `r2` the second of a two-variable table, `del1`/`del2` mean an input cell was deleted. The formulas sit outside the block: the row above it for a column-oriented table, the column to its left for a row-oriented one, the corner cell above-left for a two-variable table. A table with no room for them (block on row 1 or column A), or whose `ref` does not parse, is left alone. Each result cell is computed by giving the input cell the value from the table's edge and calculating the formulas that depend on the input, in dependency order, in an overlay the resolver consults first: the grid is never written with the substituted values, so nothing leaks into other formulas, and the table's own cells are the only cells written. In a two-variable table `r1` takes the values of the top row and `r2` those of the left column. A deleted input gives `#REF!` in every cell. Tables are calculated after the formulas, and formulas that read table cells follow in a further pass. A recalculation runs them when something was calculated or edited. Formulas that build references at run time (`INDIRECT`, `OFFSET`) are not followed from the input cell. *(pending the Excel check: which of `r1`/`r2` takes the top row, the corner position, one-variable tables with several formulas; workbooks `datatable-column`, `datatable-row`, `datatable-2d`)* ## Goal Seek `Workbook.GoalSeek(setCell, toValue, changingCell, options)` returns a `GoalSeekResult` (status `Found`, `NotFound`, `SetCellNotFormula`, `ChangingCellIsFormula` or `SetCellNotNumeric`; iterations; the value tried last or best). It calculates the workbook first, then searches in the same overlay as data tables, so the grid is untouched until the end. The search is a secant iteration from the changing cell's value; it stops when the formula cell is within the maximum change of the target (`iterateDelta`, default 0.001) or after `iterateCount` evaluations (default 100): Microsoft documents Goal Seek as using those two options. A found value is written to the changing cell and the workbook recalculated unless `Apply` is false; a failed search changes nothing. Excel does not document its search method, so the iteration count for a given problem is ours. Formulas reached through `INDIRECT` or `OFFSET` are not followed. ## Other workbooks `externalLink` parts are read into `Workbook.ExternalLinks` (index, target as recorded, sheet names, cached cells from `sheetDataSet`); `[1]Sheet1!A1` in a formula is link 1 (ECMA-376 Part 1 §18.14). A part that is missing or malformed is the defect `xlsx.external-link-unreadable`, and the link keeps its number with no cache. Default-deny: with no resolver nothing outside the file is opened, and a reference reads the cache. A single cell must be in the cache to answer; a block answers with the cached cells it has and blank for the rest, unless it has none, which is `#REF!`. A link or sheet the file does not list, a 3-D external reference and an external defined name are `#REF!`. Whole-column and whole-row blocks are cut to the cached extent. `RecalcOptions.ExternalWorkbooks` supplies an `IExternalWorkbookResolver`: it is asked once per link per calculation for a `Workbook`, and the values read are that workbook's stored values (it is not recalculated). A resolver that returns null, or throws `OfficeException` or `IOException` (reported as `calc.external-resolve-failed`), leaves the cache in charge. Values read live replace the link's cache, and the part's `sheetDataSet` is rewritten at the end of the calculation (the rest of the part is untouched); without a resolver the part is never rewritten. The resolver owns the workbooks it returns. `RecalcResult.ExternalResolved` and `ExternalMissing` count the reads. *(pending the Excel check: blank cells in cached blocks, what Excel writes for blank cached cells, `#REF!` for a cell its cache lacks; workbook `external-cached`)* ## Web functions `WEBSERVICE` and `FILTERXML` are registered and answer only through `RecalcOptions.WebServices` (`IWebServiceResolver`). `WEBSERVICE` gives `#VALUE!` for an address that is empty, over 2048 characters or not http/https, and for a response over 32767 characters (Microsoft's `WEBSERVICE` page). Without a resolver both give Excel's offline result `#VALUE!`, and a cell whose formula called one keeps its cached value when it has one (nothing is overwritten with `#VALUE!`). With a resolver that returns null the function gives `#VALUE!` as usual. *(pending the Excel check: that Excel keeps cached values for offline calls rather than showing `#VALUE!`; workbook `web-offline`)* ## Dynamic arrays and spills A formula is a dynamic array when it is stored as an array formula (`t="array"`) on a cell with the dynamic-array metadata (`cm`, [MS-XLSX] CT_DynamicArrayProperties). `Cell.SetDynamicArrayFormula` writes that form (and adds `xl/metadata.xml` when the workbook has none; a workbook whose metadata part has no dynamic-array entry is refused). Array formulas without `cm` are legacy: they fill the range they name and pad a smaller result with `#N/A`. Plain formulas get implicit intersection. A dynamic result larger than one cell is written into the grid: the anchor keeps the formula and `ref` (the spilled range), the other cells hold plain cached values, which is how Excel stores a spill. It is `#SPILL!` (and `ref` shrinks to the anchor) when a cell in the range other than the formula's own earlier spill holds a value or formula, when it touches a merged range or a table, or when it runs past the last row or column. A `#SPILL!` formula is tried again on every call, and so is a formula whose spill range was edited. If two spills overlap, the one calculated first (cell order within a level, lower levels first) wins. `A1#` reads the spill range of the anchor at `A1` (`#REF!` when `A1` is not a dynamic-array formula), and formulas that read spilled cells follow a changed spill in a further pass (at most 8). *(pending the Excel check: every blocker kind, the overlap rule, `A1#` on a one-cell result; workbooks `spill-*`)* ## After a calculation Cached values in the store are updated; formulas whose result did not change are not rewritten. A text result is kept as a formula string result. Saving writes them as it writes any edit. `fullCalcOnLoad` and the calculation chain are handled as they are for any formula edit. If a call is cancelled, some cells hold new values and some old ones; call again to finish. ## Tools `onedocument recalc [-o out] [--full] [--threads n]` calculates and saves (in place when no `-o`). `tools/CalcSweep PATH...` is the cached-value harness: it remembers every formula's stored value, calculates everything, and compares (numbers at 15 significant digits, text, booleans and errors exactly), classifying each difference as `unimplemented` (the formula calls a function the registry lacks), `external`, `downstream-or-bug` (the workbook has an unimplemented function elsewhere) or `bug`. The figure to quote is matches over the formulas whose functions are all implemented. `CalcSweep gen DIR` writes the Excel test workbooks, `CalcSweep compare ANSWERS` compares them with the oracle's cells-mode answers. ## Not here yet Pivot refresh, `NOW`/`TODAY` clocks, and the function families beyond the evaluator's seed set (a formula calling a missing function evaluates to `#NAME?`). ## Deviations - A whole-column reference as an array has the used extent's shape (above). - A relative reference inside a defined name is read as written, from cell A1. # Pivot caches and tables Namespace `RecursiveAI.OneDocument.Spreadsheets.Pivots`, in the `.Spreadsheets` package. Entry points: `Workbook.PivotCaches`, `Worksheet.PivotTables`, `PivotTable.Refresh`, `Workbook.RefreshPivots`, and `GETPIVOTDATA` in formulas. Sources: ECMA-376 Part 1 §18.10 (`pivotCacheDefinition`, `pivotCacheRecords`, `pivotTableDefinition`) and the shape of the files Excel writes. Where the schema is silent the choice below is the first reading; those marked *(pending the Excel check)* are settled by the sample workbooks in `testdata/pivots`, which are generated and committed unanswered and run through Excel once, at the gate. ## What is read, and what stays as it was The cache definition, its records part and each pivot table definition are kept as markup trees over their parts, so unknown elements and attributes, prefixes and `extLst` stay where they are. A workbook that is opened and saved without a refresh writes every pivot part byte-identical. The parts are read when first used: `Workbook.PivotCaches` (from the `pivotCaches` element and its relationships) and `Worksheet.PivotTables` (from the sheet's `pivotTable` relationships). A part that is missing or is not the expected element is reported as a defect and left as stored (`xlsx.pivot-*`, `defects.md`). `PivotCache` exposes the id, the source (kind, sheet, range, name), `RefreshOnLoad`, `SaveData`, the record count and the field names. `PivotTable` exposes the name, cache, location, the row, column and page field indexes (`-2` is the data fields) and the data fields (function, show-values-as, base field and item, number format). ## Refresh `PivotTable.Refresh()` and `Workbook.RefreshPivots()` rebuild the cache from its worksheet, table or defined-name source, then every table that uses the cache (the cache's shared item indexes change, so a sibling table cannot be left alone). The work is done in memory first: a cancelled refresh, a source that cannot be read and a cache that is not refreshable change nothing. The result lists one outcome per table: | Status | When | |---|---| | `Refreshed` | the cache and the table were rebuilt | | `NotSupported` | the cache source is external, a cube, the Data Model, a consolidation or a scenario (kept as stored), or the table has no cache | | `SourceInvalid` | the source sheet, range, table or name is missing, or a header cell of the source is empty | Refreshing writes the cells of the table into its sheet, so formulas that read them see the new values at the next recalculation; refresh itself recalculates nothing. Nothing in the written bytes depends on the clock: `refreshedDate` and `refreshedBy` change only when the options give them. ### The cache Source columns become cache fields, matched to the stored fields by name (case-insensitive); a header repeated in the source is read as `Name2`, `Name3` as Excel does. Calculated fields and group fields (`databaseField="0"`) follow the columns in their stored order. If the columns changed, every reference in the table (`rowFields`, `colFields`, `pageFields`, `dataFields`, `fieldGroup base`) is remapped; a reference to a column that is gone is dropped with a defect. - **Values.** Numbers, text, booleans, errors, dates (a number whose cell format is a date format) and blanks are read from the cells. A pivot cache lists text without regard to case: "a" and "A" are one item and the first spelling is kept. - **Shared items.** A field lists its items (`sharedItems` with `s`/`n`/`b`/`e`/`d`/`m`, in order of first appearance, and `` in the records) when it holds anything but numbers and blanks, is on an axis of any table, or is grouped. A field of only numbers (and blanks) that is not on an axis stores them inline (``, ``) and its `sharedItems` carries only the flags and `minValue`/`maxValue`. *(pending the Excel check)* - **Flags.** `containsSemiMixedTypes`, `containsNonDate`, `containsDate`, `containsString`, `containsBlank`, `containsMixedTypes`, `containsNumber`, `containsInteger`, `minValue`/`maxValue`, `minDate`/`maxDate`, written only where they differ from the schema default (ECMA-376 Part 1 §18.10.1.90). - **Records.** One `r` per source row; calculated and group fields have no entry. A cache that stores no records (`saveData="0"`) still stores none. A records part that did not exist is added, with its relationship and the definition's `r:id`. - `recordCount` is updated. ### Grouping A cache field with `fieldGroup` is recomputed from the base field's shared items. | `groupBy` | Group items | Notes | |---|---|---| | `years` | `end` | | | `quarters` | `end` | | | `months` | `end` | years are merged unless a years group is also on the table | | `days` | `end`; with `groupInterval` > 1, one item per interval `d-Mon - d-Mon` | | | `hours`, `minutes`, `seconds` | `end` | label text *(pending the Excel check)* | | range (no `groupBy`) | `end` | `lo-hi` with `hi = lo + interval - 1` when the data and interval are whole numbers, else `lo + interval` *(pending the Excel check)* | | discrete (`discretePr`) | the stored group items | stored members follow their item by value; an item that is new gets a group of its own named after it | `end` use `M/d/yyyy` (the en-US short date) *(pending the Excel check for other locales)*. With `autoStart`/`autoEnd` (the default) the `startDate`/`endDate` or `startNum`/`endNum` are set to the smallest and largest value in the data. A record whose base value is blank has no group and is left out of the table. A `groupBy` the library does not know drops the group with `pivot.group-unsupported`. ### The table Each table is read with its field indexes mapped to the new cache, then laid out. - **Field items.** An axis field's items (`pivotField/items`) keep their stored order, their flags (`h`, `s`, `n`, …) and the trailing subtotal items (`t="default"`, added when `defaultSubtotal` is on and there are none); items whose value is gone are dropped. New items are placed by `sortType`: `ascending` and `descending` sort the whole list afresh (numbers and dates, then text without regard to case, then booleans, errors, blanks; grouped fields by group order); `manual` appends the new items in ascending order. The item `h="1"` hides an item, and its records are left out of every total. - **Axes.** Only combinations with records are listed. A table with several data fields lists them as the field `-2` on the column axis (added when the stored definition has none; `-2` anywhere else is kept where it is). - **Aggregates.** Sum, Count (non-empty values), Average, Max, Min, Product, CountNums, StdDev, StdDevP, Var and VarP over the numbers of the group; text, booleans and errors are ignored by all but Count. Average, StdDev and Var of nothing (and StdDev/Var of one value) give `#DIV/0!`. A group that has no records at all is an empty cell, or `missingCaption` when `showMissing` is on and one is set. - **Show values as.** Normal, percent of grand total, of row and of column, of parent row and parent column, index, and difference, percent, percent difference and running total along a base field (`baseItem` an item position, `1048828` the previous and `1048829` the next item). The base-field forms apply at the base field's level; elsewhere the cell is empty. Percent of parent, percent running total and the rank forms show the normal value with `pivot.show-data-as-unsupported`. Percent forms get `0.00%` unless the data field names a number format. - **Calculated fields.** The formula is read by the calc engine with each field name (also in single quotes) standing for the sum of that field over the cell's group; the result is the formula applied to those sums, never the sum of per-record results (Excel's rule for calculated fields). An unreadable formula shows `#VALUE!` with `pivot.calculated-field-unparsable`. - **Report filters.** The selected item (`pageField item`) follows its value through the refresh; with none the field shows `(All)`, or `(Multiple Items)` when the field allows several and hides some. The records are filtered accordingly. - **Not applied.** Label, value and date filters (`filters`) are kept as stored and reported with `pivot.filters-not-applied`; custom subtotal functions show the automatic subtotal (`pivot.custom-subtotals-unsupported`). ### Layout The table keeps its top-left cell; the page fields go in the rows above it, one per row with a blank row between them and the table (the table moves down when there is no room). The old cells (location and page fields) are cleared, then the new ones written. Numbers are written as numbers (item labels that are numbers or dates too, dates with `m/d/yyyy`), text as text; compact-form children are indented with the cell alignment, one level per nesting level times `indent`. Fills, fonts and borders come from the table style, as in Excel. - **Compact** (`compact` true on the field, the default): consecutive compact row fields share the first column, captioned `Row Labels` (`rowHeaderCaption`); each item has its own row, its subtotal in that row when `subtotalTop` is on (default), else on a `X Total` row after its children. - **Outline** (`compact="0"`): one column per field, captioned with the field name; items and their subtotals as in compact. - **Tabular** (`compact="0" outline="0"`): one column per field, a row per innermost combination with the outer labels on the first row of their group, subtotal rows `X Total` after the group. - **Columns.** Captioned `Column Labels` (the field name when the first column field is not compact) in the first header row; one header row per column field, each label over the first column of its group; subtotals `X Total` after a group; the `Grand Total` column after all. With several data fields the data field names are the innermost header row, and totals are `Total `. - **Grand totals.** `colGrandTotals` (default on) is the Grand Total row; `rowGrandTotals` the Grand Total column *(pending the Excel check for which attribute is which)*. Subtotals appear for fields with `defaultSubtotal` (default on) that have a field inside them. - **Location** (`firstHeaderRow`, `firstDataRow`, `firstDataCol`): `firstDataCol` is the number of label columns; `firstDataRow` is the column field count (the data-field row included) plus one; `firstHeaderRow` is 1 when there are column fields, else 0 for several data fields and 1 for one *(pending the Excel check)*. - **`rowItems` and `colItems`** list the rows and columns: `x v` is the item's position in its field's item list (the data field's index for the data row/column, which also sets `i`), `r` repeats that many leading entries of the previous line, `t="default"` marks a subtotal and `t="grand"` a grand total. - A data field placed before the innermost field, and other unusual positions of `-2`, are laid out with the data field names as extra levels; the result is not compared with Excel. ## GETPIVOTDATA `GETPIVOTDATA(data_field, pivot_table, [field1, item1], …)` reads what the table shows, like Excel: the stored `rowItems`/`colItems` give each row and column its path, and the answer is the cell at that row and column. So it answers from the last refresh; after the source changes it agrees with the cells until the next `Refresh`, then at the next recalculation. - `data_field` is a data field's caption (`Sum of Sales`) or its source field's name (`Sales`). - `pivot_table` is any cell of the table's body; a cell outside every table, or on a sheet without one, gives `#REF!`. - The field/item pairs may come in any order; the row fields named must be the first fields of the row axis (and likewise the columns), or the result is `#REF!`. No pair for an axis gives its grand total. Item matching ignores case; a number matches a numeric item. - A report-filter field answers only for the item it currently shows. - An unknown field or item, a combination the table does not show, an empty cell and an odd number of pair arguments give `#REF!`. The function is `ICalcResolver.GetPivotData` in core; a resolver with no pivot tables returns `#REF!`. ## Not supported Pivot charts, slicers, timelines, the Data Model, OLAP and external sources are kept as stored. The refresh of calculated items, label and value filters, show-values-as parent/rank forms, custom subtotals and the hierarchy-based (OLAP) fields is reported, not reproduced. # Calc: values, formulas, evaluator and function registry Namespace `RecursiveAI.OneDocument.Calc` in the core package. This page is the contract the function families, the workbook binding, recalculation and the template engine build on. Excel is the specification for anything the text below does not settle; rules settled by a run against Excel 16.0 (build 20527) say so and cite the committed grid in `testdata/calc/`; a rule marked *(unverified)* is a first reading and is listed at the end. ## Values `CalcValue` is a 24-byte struct: a kind tag, a `double` and an object slot. | Kind | Holds | Allocation | |---|---|---| | `Blank` | an empty cell. `CalcValue.Missing` is a blank that remembers it was an omitted argument (`IsMissing`). | none | | `Number` | a `double` (dates and times are serials). Never NaN or infinite: a result that is not finite is `#NUM!`. | none | | `Text` | a `string` | the string | | `Boolean` | TRUE or FALSE | none | | `Error` | a `CalcError`: `#NULL! #DIV/0! #VALUE! #REF! #NAME? #NUM! #N/A #GETTING_DATA #SPILL! #CALC! #BLOCKED! #CONNECT! #BUSY! #FIELD! #UNKNOWN!` | none | | `Array` | a `CalcArray`: rows × columns, row-major, cells are scalars. Dense, or one value broadcast over a shape (`CalcArray.Broadcast`). No nested arrays. | the array | | `Reference` | a `CalcReference`: one or more `CellArea`s (sheet name or null for the formula's sheet, optional last sheet for 3-D, 1-based inclusive rows and columns), optionally `IsSpill` for `A1#`. Not read until something reads it. | the reference | | `Lambda` | a `LambdaValue` (parameter names, body, captured names) | the lambda | `CalcValue.Equals` is structural for scalars and identity for the rest. Comparison as the `=` and `<` operators do it is `Evaluator.Compare`. `CalcConvert` holds Excel's scalar conversions: `ToNumber` (blank 0, TRUE 1, text read as number, percent, currency amount, date or time), `ToBoolean`, `ToText` (up to 15 significant digits), `NumberToText`, `TextToNumber`. ## Formula text `FormulaParser.Parse` / `TryParse` read formula text (no leading `=` unless `LeadingEquals`) into a `FormulaNode` tree; `FormulaPrinter.Print` writes it. * **Notations.** A1 and R1C1 (`FormulaParserOptions.Syntax`, with the cell the formula sits in as origin: relative R1C1 offsets are resolved against it). In the tree a coordinate is a `RefCoord(index, absolute)` and a relative index is an absolute position in the formula's own frame, so one tree prints as A1 or R1C1. * **Forms.** `FormulaForm.File` is what is stored: en-US separators and names, `_xlfn.` / `_xlfn._xlws.` prefixes, `@` as `_xlfn.SINGLE(...)`, `#` as `_xlfn.ANCHORARRAY(...)`. `Display` is what a user reads: no prefixes, `@x`, `A1#`, local separators and names from a `FormulaLocale`. * **Round trip.** A tree that was parsed and not edited prints back to the characters it came from in file form: nodes keep their spaces (`LeadingSpace`, `TrailingSpace`, the operator text), the spelling of numbers, references, names, function names (with prefixes), error literals and array constants. `FunctionCallNode.NameIsFileForm` is false for text a user typed (`FileForm = false`), where printing to file form adds the prefix `FutureFunctions.PrefixFor(name)` gives. * **What parses.** Numbers; text with doubled quotes; TRUE/FALSE; error literals (all fifteen); cells, areas, whole rows and columns, with `$`; sheet prefixes (`Sheet1!`, `'My Sheet'!`, `Sheet1:Sheet3!`, `[1]Sheet1!`, `[1]!Name`, `Sheet1!#REF!`); defined names (dots and `_xlpm.` allowed); table references (`Table1[Col]`, `Table1[[#This Row],[Col]]`, `[@Col]`; the specifier is kept as written and `GetItems()` splits it); function calls with omitted arguments (`MissingNode`); calls of computed callees (`LAMBDA(x,x)(1)`, `InvokeNode`); array constants; operators with Excel's precedence, lowest first: comparison, `&`, `+ -`, `* /`, `^` (left associative), `%` postfix, prefix `+ - @`, then reference operators `:` (range), space (intersection), `,` (union, only inside parentheses). Unary minus binds tighter than `^`, so `-2^2` is 4 as in Excel. * **Template extensions** (ours, off unless `TemplateExtensions`): a dotted name `item.price` becomes `MemberPathNode`, resolved through `ICalcResolver.ResolveMember`. Array lifting needs no syntax: a member path may return an array and the evaluator lifts operators and functions over it. In normal mode `item.price` is an ordinary defined name (names may contain dots). * **Locales.** `FormulaLocale` carries the decimal, argument, array-column and array-row separators and an optional function-name table. `FormulaLocale.ForDecimalSeparator(",")` gives the comma-decimal family (`;` arguments and rows, `.` array columns); `ForDecimalSeparator(",", '\\')` is what Excel 16 wrote and accepted when its separators were set to a comma decimal on an en-US system (array columns `\`; the same with a space as thousands separator). 52 formulas agree with `Formula2` and `Formula2Local` in that configuration (`testdata/calc/excel-grid/loc*.fgrid`). Separators of the regional installs, and **localized function-name tables, are not shipped**: the only published per-language lists found are Microsoft's localized function pages, which are machine-translated and not reliable (e.g. `CEILING.MATH` becomes `DECKE. MATHEMATIK`, `ABS` becomes `АБС-пластик`), and the laptop's Excel is en-US only, so they cannot be checked. A `FormulaLocale` accepts any table the caller supplies. * **Errors.** `FormulaParseException` (an `OfficeFormatException`, `DefectKind.Formula`) or `TryParse` with a `FormulaSyntaxError(code, message, offset, length)`; the codes are in `defects.md`. Text longer than `OfficeLimits.MaxFormulaLength` or nested deeper than `MaxFormulaDepth` throws `OfficeLimitException` from `Parse` and is `formula.too-long` / `formula.too-deep` from `TryParse`. * **Shared formulas.** `FormulaTranslator.Translate(anchorTree, rowDelta, columnDelta)` moves relative coordinates; a reference pushed off the sheet becomes `#REF!`; unchanged subtrees are shared. * **Positions.** Every parsed node has `Offset` and `Length` in the source text. ## Evaluator `new Evaluator(resolver, registry?, limits?, options?)`; `Evaluate(tree, CalcCell origin)` returns a scalar, an array or a lambda, never a reference (a formula that is only `A1` returns that cell's value). One evaluator per thread. Opening a workbook never calls it; recalculation does. * **Resolver.** `ICalcResolver` has one required method, `GetCell(sheet, row, column)`. Defaults, overridable: `GetRange` (loops `GetCell`), `TryExpandSheets` (3-D; default none, so `#REF!`), `ResolveName` (`#NAME?`), `ResolveStructured` (`#REF!`), `ResolveExternal` (`#REF!`: external links are default-deny), `TryGetSpill` (none, so `A1#` is `#REF!`), `ResolveMember` (`#NAME?`), `Clock` (the time `TODAY`/`NOW` read; null falls back to `EvaluatorOptions.Clock`, then the system clock). A resolver may return references from `ResolveName` / `ResolveStructured` (names for ranges, tables). The sheet passed is the reference's sheet or the origin's. * **Operators.** Errors propagate left first. Arithmetic coerces operands with `CalcConvert.ToNumber` (`#VALUE!` when text is not a number); results that are not finite are `#NUM!`; `x/0` is `#DIV/0!`; `0^0` is `#NUM!`, `0^negative` is `#DIV/0!`, negative base with a fractional exponent is `#NUM!`. Comparison orders numbers < text < booleans, text ignoring case, blank as 0 / "" / FALSE against the other kind, numbers equal when they agree to 15 significant digits. `&` converts with `ToText`. Operators lift over arrays: shapes broadcast (a one-row or one-column array stretches, a scalar stretches), a cell outside a shorter array is `#N/A`. * **Near-zero snap** (measured: `testdata/calc/excel-grid`, probes `pr`). A `+` or `-` that is the outermost operation of the formula, and `SUM`'s additions, return 0 when the result is smaller than 2^-49 of the binade of the larger operand, that is fewer than 8 units in the last place of that operand: `=1+7ulp-1` is 0, `=1+8ulp-1` is not (checked at binades 2^-4, 2^-2, 2^0, 2^6 and 2^9). So `=1.1-1-0.1` and `=(1+1E-15)-1` are 0, `=SUM(1.1,-1,-0.1)` and `=SUM(0.1,0.2,-0.3)` are 0, but `=(1.1-1-0.1)` is 8.3E-17, `=IF(TRUE,1.1-1-0.1)` is 8.3E-17 and `=1.1-1` is 0.10000000000000009. Only the outermost operation snaps; any wrapper (parentheses, `IF`, `ABS`, `IFERROR`) leaves the raw result. * **Power.** Negative base with a fractional exponent: the real root when the exponent is 1/odd (`(-8)^(1/3)` is -2), otherwise `#NUM!`. A negative base with an integer exponent of 2^32 or more is `#NUM!` (`(-1)^2147483649` is -1, `(-1)^4294967296` is `#NUM!`). `Math.Pow` differs from Excel's routine in the last digit or two for some inputs (`"12:30"^"1,000"`, 14th digit); not reproduced. * **Number to text** (`&`, `CONCAT`, `TEXT` with General). 15 significant digits; plain digits up to 1E+19 and scientific from 1E+20 (`1.5E+20`); below 1, plain while the plain form needs at most 18 decimals, so `1E-18` and `1.5E-17` are plain, `1E-19` is `1E-19` and `1.23456789012345E-5` is `1.23456789012345E-05`. * **Accepted by Excel only at the formula bar.** Excel refuses at entry: `SUM()`, `AND()`, a constant that overflows (`1E308*10`), `LET` with a repeated name, `#SPILL!` / `#CALC!` typed as literals, a leading `==`. Our parser accepts these and evaluates them (`SUM()` is `#VALUE!` here); the grid lists them in `mismatches.txt` as entry rejections, not value differences. * **References.** Range `:`, intersection and union operate on references and give references (`#NULL!` for an empty intersection, `#VALUE!` for non-references). `A1#` asks the resolver for the spill extent. * **Arrays and implicit intersection.** By default a range where one value is wanted makes the result an array (dynamic arrays). `EvaluatorOptions.LegacyImplicitIntersection` cuts it to the formula's row or column instead; `@x` and `SINGLE` do so explicitly. The caller gets the array; putting it on a grid and `#SPILL!` detection belong to recalculation. * **LET / LAMBDA.** Names live in an immutable scope list; a lambda captures the list at creation. A lambda is invoked by `f(args)` (a LET name or parameter), by `LAMBDA(...)(args)`, by `Evaluator.Invoke`. Recursion deeper than `OfficeLimits.MaxRecursionDepth` yields `#NUM!` and sets `RecursionLimitHit`. * **Limits.** `EvaluatorOptions.MaxSteps` (node visits, plus one per element when a function or operator is lifted over an array and the cells that call returns or points at; default 50,000,000) and `OfficeLimits.MaxAllocationBytes` (arrays, 24 bytes a cell) throw `OfficeLimitException`. Function code makes arrays with `Evaluator.NewCells` so the ceiling applies. * **Allocation.** Scalar arithmetic and comparison over literals allocate nothing per evaluation (tested). References are cached on their node; function calls use a value stack, not per-call arrays; lifting, arrays and text results allocate what they hold. ## Function registry `FunctionRegistry` maps names (case-insensitive, `_xlfn.` / `_xlws.` prefixes ignored) to `FunctionDefinition`s. `FunctionRegistry.CreateDefault()` returns a fresh registry with the library's functions; a function family adds one `Register…` method that calls `Add` for each function. Nothing is found by reflection; the evaluator is never edited to add a function. `FunctionDefinition.Create(name, minArgs, maxArgs, kinds, implementation, flags, repeatLast)`: * **Argument kinds** (`ArgKind`), one per position, the last repeating with `repeatLast`: * `Value`: one scalar. A one-cell reference is read; a block or array makes the call *lift*: the function runs once per element and the evaluator assembles the array result (shapes broadcast). Errors in `Value` arguments return the first error left to right without calling the function, unless the function has `HandlesErrors`. * `Array`: values as an array (references are read; a scalar stays a scalar; lambdas pass through). No lifting. * `Any`: the argument as evaluated: references stay references (so a function can tell `A1` from `5`, or take a shape), arrays and lambdas pass through. * **Implementation** `EagerFunction(Evaluator, ReadOnlySpan args)`; the span is valid during the call. An omitted argument arrives as `CalcValue.Missing`. Return a scalar, an array (a dynamic-array result) or a reference. Helpers on the evaluator: `Dereference`, `ToScalar`, `ToArray`, `ImplicitIntersect`, `TryReadAreas`, `NewCells`, `Invoke`, `Origin`, `Options`. * **Lazy functions** `CreateLazy`: `LazyFunction(Evaluator, ReadOnlySpan args)` for `IF`, `IFERROR`, `LET`: evaluate only what is needed with `Evaluate(node)`. * **Flags**: `Volatile` (sets `Evaluator.UsedVolatile`), `HandlesErrors`, `UsesOrigin`. * **Arity**: too few or too many arguments is `#VALUE!`. * **Custom C# functions**: `registry.AddCustom(name, min, max, args => value, isVolatile)`: value arguments that lift over arrays, errors propagate before the call, delegate-based and AOT-safe. Shipped now (the seed set that proves the contract): `SUM AND OR NOT CONCAT LEN ISERROR ROWS COLUMNS INDEX SEQUENCE MAKEARRAY MAP TEXT IF IFS CHOOSE IFERROR IFNA LET LAMBDA SINGLE ANCHORARRAY`. `TEXT` runs on the number-format service. The lookup, dynamic-array and LAMBDA-helper families are in `docs/calc-functions.md`; the other families come as their own units. A registry function written bare (`REDUCE(0, a, SUM)`) works as a lambda in the helpers that take one. ## Future-function prefixes `FutureFunctions.PrefixFor(name)`: the set is the `future-function-list` and `worksheet-only-function-list` rules of the Excel extensions to the file format (Formulas grammar), corrected by Excel 16.0: each of ~500 candidate names (the grammar plus Microsoft's alphabetical function list) was entered with 0..4 arguments and the saved file read back. Excel stores `NETWORKDAYS.INTL`, `WORKDAY.INTL`, `ISO.CEILING` and `ECMA.CEILING` bare although the grammar lists them, and prefixes `IFS`, `CONCAT`, `TEXTJOIN`, `SWITCH`, `MAXIFS`, `MINIFS`, `XLOOKUP`, `XMATCH`, `GROUPBY`, `REGEX*`, `TRIMRANGE` and others the grammar does not list. `FILTER` and `SORT` get `_xlfn._xlws.`. `@` on a range is not stored (`=@A1:A3` is saved as `A1:A3`); `_xlfn.SINGLE` appears only for an explicit `SINGLE`. Names the grammar lists that this build does not know (`COPILOT`, `LONGTEXT`, `PQSOURCE`, `PYTHON_*`, `QUERYSTRING`) follow the grammar. Answers: `testdata/calc/future-functions-excel.tsv`. ## Not here yet Dependency graph, recalculation, putting spills on the grid, `#SPILL!` detection, workbook names / tables / sheets / external links (the resolver is the seam), localized function-name tables, BIFF Ptg and XLSB binary formulas, OpenFormula, the function families beyond the seed set. ## Open against Excel *(unverified)* Text-to-number coercion for unusual spellings (fractions, parentheses); text ordering beyond the Windows-like word sort already matched on the grid; array separators and function names of regional Excel installs; how `INDEX` treats a missing row with a one-column range. # Calc: function families The functions registered by `FunctionRegistry.CreateDefault()` for these four families, written from Microsoft's per-function reference pages. Columns: **status** is `docs` when the behaviour follows the reference page and its examples (covered by `FunctionFamilyTests`), `pending` when the reference leaves an edge open and the choice made here waits for the Excel grid (`testdata/calc/function-grid`, answers not yet recorded); the lookup, array and LAMBDA families have their own grid in `testdata/calc/lookup-grid`. Function arguments that are `Value` lift over arrays and propagate errors before the call (see `calc.md`); the aggregate functions take references and arrays unchanged. Numbers go through the core rules: 15-significant-digit comparison, `Evaluator.AddSnapped` for sums, `CalcConvert` for text/number coercion. ## Shared helpers * **Criteria** (`CalcCriteria`, public): the matching rule of the `*IF(S)` family. A criterion is a number, boolean, error, or text with an optional operator (`=`, `<>`, `<`, `<=`, `>`, `>=`). Equality on text understands `?`, `*` and `~` escapes and ignores case; an operand that reads as a number compares numerically against number cells (and, for equality, text cells that read as the same number); `""` matches blank and empty-text cells, `"="` only blank cells, `"<>"` every non-blank cell. A blank criterion cell counts as 0 *(pending)*. * **Resolver hooks** (all additive, defaults say "not available"): `IsFormula`, `GetFormulaText`, `GetRowVisibility`, `IsSubtotalCell`, `GetCellInfo`, `GetSheetIndex`, `GetSheetCount`, `GetFileName`, `GetInfo`. * **Random numbers** come from `EvaluatorOptions.Random` (a seeded `Random` gives repeatable results); `OfficeLimits.RegexMatchTimeoutMilliseconds` bounds regular-expression matches. ## Math and trigonometry | Functions | Status | Notes | |---|---|---| | `ABS SIGN SQRT SQRTPI EXP LN LOG LOG10 POWER PI DEGREES RADIANS` | docs | `POWER` shares the `^` operator rules (`0^0` is `#NUM!`). `LOG` base 1 is `#DIV/0!`. | | `ROUND ROUNDUP ROUNDDOWN TRUNC INT` | docs | Rounding works on the 15-digit decimal value, so `ROUND(2.675,2)` is 2.68. | | `MROUND CEILING FLOOR CEILING.MATH FLOOR.MATH CEILING.PRECISE FLOOR.PRECISE ISO.CEILING EVEN ODD` | docs | The quotient is rounded to 15 digits before the step is chosen, so `CEILING(0.3,0.1)` is 0.3. `FLOOR` with a negative number and positive significance is *(pending)*. | | `MOD QUOTIENT` | docs | `MOD` keeps the divisor's sign. Very large quotients (Excel's old 2^27 limit) *(pending)*. | | `SIN COS TAN ASIN ACOS ATAN ATAN2 SINH COSH TANH ASINH ACOSH ATANH COT COTH CSC CSCH SEC SECH ACOT ACOTH` | docs | `ACOT` is in (0, pi). Huge arguments (|x| >= 2^27) are computed, not refused *(pending)*. | | `FACT FACTDOUBLE COMBIN COMBINA PERMUT PERMUTATIONA MULTINOMIAL GCD LCM` | docs | Arguments are truncated. `GCD`/`LCM` refuse values from 2^53. `PERMUT`/`PERMUTATIONA` are statistical in Microsoft's list and registered here because they share the factorial table. | | `ROMAN ARABIC BASE DECIMAL` | docs | `ROMAN` forms 1-4 use the tables in `MathNumbers.cs` checked only against the reference's `499` example *(pending: the full 1-3999 range per form)*. `ARABIC` accepts non-canonical numerals *(pending)*. | | `SUM PRODUCT SUMSQ SUMPRODUCT SUMX2MY2 SUMX2PY2 SUMXMY2 SERIESSUM` | docs | `SUM` is the seed implementation. Direct arguments are coerced; cells in references and arrays contribute only numbers. | | `SUMIF SUMIFS` | docs | Ranges must be references (`#VALUE!` otherwise). `SUMIF` re-anchors a differently sized sum range at its top-left cell; `SUMIFS` needs equal sizes. A criterion array returns an array. | | `SUBTOTAL AGGREGATE` | docs | Functions 1-11 skip filtered rows, 101-111 also hand-hidden rows; cells reported by `IsSubtotalCell` are always skipped. `AGGREGATE` options 0-7 and functions 1-19 as documented. Non-reference arguments to `SUBTOTAL` are treated as values *(pending)*. | | `MMULT MDETERM MINVERSE MUNIT` | docs | LU decomposition; a pivot that is zero at the matrix scale is singular (`#NUM!`). Results near zero may differ from Excel in the last bits *(pending)*. | | `RAND RANDBETWEEN` | docs | Volatile; drawn from the evaluator's seeded sequence (`EvaluatorOptions.RandomSeed`, default 0), like `RANDARRAY`. | | `SEQUENCE` | seed | | ## Logical | Functions | Status | Notes | |---|---|---| | `AND OR NOT IF IFS IFERROR IFNA` | seed | | | `XOR TRUE FALSE` | docs | `XOR` skips text and blanks in references/arrays; no logical value at all is `#VALUE!`. | | `SWITCH` | docs | Lazy; compares like `=`; an array expression switches element by element. | ## Information | Functions | Status | Notes | |---|---|---| | `ISBLANK ISNUMBER ISTEXT ISNONTEXT ISLOGICAL ISERR ISERROR ISNA ISREF` | docs | | | `ISEVEN ISODD` | docs | Truncate; booleans and non-numeric text are `#VALUE!` *(pending for booleans)*. | | `N T NA TYPE ERROR.TYPE` | docs | `TYPE` of a multi-cell reference is 64. `ERROR.TYPE` numbers for the errors after `#GETTING_DATA` follow the reference page; `#BUSY!` has no listed number and gives 15 *(pending)*. | | `ISFORMULA FORMULATEXT SHEET SHEETS CELL INFO` | docs | Through the resolver hooks. `CELL("format")` maps the format code to the short codes of the reference table. `INFO` defaults are Excel for Windows values. `CELL` without a reference uses the formula's own cell *(pending: Excel uses the last changed cell)*. | ## Text | Functions | Status | Notes | |---|---|---| | `LEFT RIGHT MID LEN REPLACE FIND SEARCH` and the `*B` forms | docs | The byte forms count one byte per character, as Excel does unless a double-byte language is the editing language *(pending: DBCS behaviour)*. `SEARCH` supports `? * ~`. Lengths are UTF-16 code units. | | `SUBSTITUTE REPT TRIM CLEAN UPPER LOWER PROPER EXACT` | docs | `TRIM` removes only U+0020. `CLEAN` removes code units 0-31. | | `CHAR CODE UNICHAR UNICODE` | docs | `CHAR`/`CODE` use Windows code page 1252; characters outside it give 63 from `CODE`. | | `CONCAT CONCATENATE TEXTJOIN ARRAYTOTEXT VALUETOTEXT` | docs | `CONCATENATE` lifts over arrays; results above 32,767 characters are `#VALUE!`. | | `TEXTBEFORE TEXTAFTER TEXTSPLIT` | docs | Delimiter arrays, negative instances, `match_end`, `if_not_found`, padding. Overlapping delimiters are not re-scanned from the right *(pending)*. | | `VALUE NUMBERVALUE FIXED DOLLAR TEXT` | docs | `VALUE("")` is `#VALUE!` *(pending)*. `DOLLAR` always uses `$`. | | `REGEXTEST REGEXEXTRACT REGEXREPLACE` | docs | PCRE2 patterns run on `System.Text.RegularExpressions` after translation: `\d \w \s \h \R \Q..\E \x{..}`, POSIX classes and `(?P)` are rewritten to ASCII semantics. Possessive quantifiers and other constructs .NET cannot parse give `#VALUE!`, as does a match over `OfficeLimits.RegexMatchTimeoutMilliseconds`. `REGEXEXTRACT` mode 1 returns a column and mode 2 a row *(pending)*. | | `ASC DBCS JIS PHONETIC BAHTTEXT` | docs | `ASC`/`DBCS` convert whatever the text holds (Excel does so only with a double-byte editing language *(pending)*). `PHONETIC` returns the cell text. | | `TRANSLATE DETECTLANGUAGE` | docs | Service functions: `#CONNECT!` offline *(pending: Excel's offline value)*. | What each family does, written from Microsoft's function reference pages. Where the pages leave a behaviour open, the code takes the reading below and the Excel grid in `testdata/calc/lookup-grid` settles it at the full Excel run; every such item is listed under **Open**. ## Lookup and reference `MATCH`, `XMATCH`, `XLOOKUP`, `LOOKUP`, `VLOOKUP`, `HLOOKUP`, `INDEX`, `CHOOSE`, `CHOOSECOLS`, `CHOOSEROWS`, `OFFSET`, `INDIRECT`, `ROW`, `COLUMN`, `ROWS`, `COLUMNS`, `AREAS`, `ADDRESS`, `TRANSPOSE`, `HYPERLINK`, `GETPIVOTDATA`, `TRIMRANGE`. * **Matching.** The type must agree: the number 1 never matches the text "1", and a blank cell matches nothing. Text ignores case; wildcards (`*`, `?`, `~` escape) apply to `MATCH` type 0, `XMATCH`/`XLOOKUP` match mode 2 and exact `VLOOKUP`/`HLOOKUP`. A looked-up blank counts as 0. * **Approximate search** is a binary search that assumes sorted data and, on unsorted data, answers whatever the probing finds. Cells of another type than the target are stepped over towards the low end. * **`XLOOKUP`** returns a reference when `return_array` is one (so `XLOOKUP(...):XLOOKUP(...)` is a range). A `lookup_array` that is not one row or one column is `#VALUE!`; a `return_array` of the wrong length is `#VALUE!`. * **`OFFSET`** needs a reference (`#VALUE!` otherwise), a zero height or width is `#REF!`, a negative one extends up or left, a result off the sheet is `#REF!`. Volatile. **`INDIRECT`** reads A1 or R1C1 text through the formula parser and accepts only a cell, a range, a defined name or a structured reference; anything else, and an external workbook, is `#REF!`. Volatile. * **`GETPIVOTDATA`** reads what the pivot table shows (see `spreadsheets-pivots.md`): the answer is the table as last refreshed, `#REF!` when the sheet has no pivot table at the cell, the data field or an item is unknown, or the combination is not shown. Through `ICalcResolver.GetPivotData`; without a workbook it is `#REF!`. **`HYPERLINK`** returns `friendly_name`, or the link text; opening the link is the host's. * **`TRIMRANGE`** removes empty edge rows and columns (modes 0 none, 1 leading, 2 trailing, 3 both). The trim-ref operators (`A1:.E8`) are not parsed yet. ## Dynamic arrays `FILTER`, `SORT`, `SORTBY`, `UNIQUE`, `TAKE`, `DROP`, `EXPAND`, `HSTACK`, `VSTACK`, `TOCOL`, `TOROW`, `WRAPROWS`, `WRAPCOLS`, `RANDARRAY`, `SEQUENCE`. * **Sort order**: numbers, then text (ignoring case, `TextCollation`), then FALSE before TRUE, then errors; descending reverses that; empty cells come last either way. The sort is stable. Several keys are given as arrays for `sort_index` and `sort_order`. * An empty result is `#CALC!`. `HSTACK`/`VSTACK`/`EXPAND` pad with `#N/A`; `WRAPROWS`/`WRAPCOLS` likewise. * `RANDARRAY` draws from the evaluator's seeded sequence (`EvaluatorOptions.RandomSeed`): the same seed gives the same numbers. * Results are arrays; putting them on the grid and `#SPILL!` belong to recalculation. ## LAMBDA helpers `MAP`, `REDUCE`, `SCAN`, `BYROW`, `BYCOL`, `MAKEARRAY`, `ISOMITTED`, `GROUPBY`, `PIVOTBY`, `PERCENTOF`. * The helpers take a function written as a LAMBDA or as a bare function name (`MAP(a, LEN)`, `REDUCE(0, a, SUM)`): a name that is not a defined name or a LET name becomes a function value. Only functions that evaluate their arguments before the call can be used this way (not `IF`, `LET`). * A lambda result that is an array of more than one cell makes `MAP`, `SCAN`, `BYROW`, `BYCOL` and `MAKEARRAY` `#CALC!` as a whole (measured on Excel 16.0 for `MAP` and `MAKEARRAY`); `REDUCE` may carry an array accumulator. * `GROUPBY` supports one aggregation function, a `field_headers` mode 0-3, total depth, sort order and a filter array; `PIVOTBY` supports one value column and grand totals. ## Open: lookup, arrays and LAMBDA helpers, pending the 1.0 Excel grid The function pages do not say, and the code takes the reading in brackets: 1. *(Settled against the 1.0 Excel grid, 99.5 % of the lookup grid agrees; what is left is not a search.)* `XMATCH` binary modes bisect across types in the order numbers, text, logical values, errors (an equal probe is kept and the search goes on towards the start when ascending, the end when descending); `XMATCH` next-smaller/larger scans compare across types and an empty needle is the empty text; legacy approximate `MATCH`/`VLOOKUP`/`LOOKUP` probes step up to a cell of the target's type and stop on an equal cell, taking the last of the equal run when ascending; `MATCH` type -1 over a short array is a linear read. Open: how Excel handles `MATCH` type -1 beyond short arrays (the code bisects), and where the question mark sorts among the other symbols (it sorts after the asterisk). 2. `MATCH(x, a, )` and `VLOOKUP(x, t, i, )` with an empty argument between commas (0 / FALSE). 3. *(Settled.)* A binary `XMATCH`/`XLOOKUP` exact search returns the first of equal cells when ascending and the last when descending. 4. `XLOOKUP` with an array `lookup_value` whose matches are rows or columns (each match is reduced to its first cell by the lifting rule). 5. `ROW`/`COLUMN` of an array or constant (`#VALUE!`), and of a union (first area). 6. `ROWS`/`COLUMNS` of a union. 7. Errors, blank cells and mixed types in `SORT` (errors after text and logicals, blanks last in both directions); `SORT` with fewer orders than keys (the last order repeats). 8. `UNIQUE` equality for numbers that differ in the 16th digit, and blank against 0 (distinct). 9. `TAKE`/`DROP` with neither count given (`#VALUE!`); `FILTER` when `include` holds text (`#VALUE!`) or non-integers (non-zero is true). 10. `RANDARRAY` with integer bounds that are not whole numbers (`#VALUE!`). 11. The optional `[y]` parameter syntax of `LAMBDA` and how `ISOMITTED` sees a skipped argument; the parser does not read `[y]` yet. 12. `GROUPBY`: "automatic" `field_headers`, labels of subtotal rows, layout of several row fields (repeated outer keys), `field_relationship`, `HSTACK` of several functions; `PIVOTBY`: several value columns, total depths beyond 1, `relative_to`, the text of cells with no data (empty text). 13. `INDIRECT` with leading or trailing spaces, `$` signs, unions and intersections; `OFFSET` with array offsets. 14. `TRIMRANGE` of a range with nothing in it (`#REF!`) and the trim-ref operators. 15. `ADDRESS` quoting of sheet names that look like cell references or numbers. ## Financial, engineering and database Behaviour of the three families registered by `FunctionRegistry.CreateDefault()`. Rules marked *(open)* are first readings that the committed grids (`testdata/calc/function-grids/`) settle against Excel at the 1.0 gate. Behaviour of the three families registered by `FunctionRegistry.CreateDefault()`. Excel is the specification; everything below was built from Microsoft's published function reference. Rules marked *(open)* are first readings that the committed grids (`testdata/calc/function-grids/`) will settle against Excel. ### Conventions shared by all three * Arguments are `Value` arguments unless said otherwise: a block or array makes the call lift and return an array, an error argument is returned before the call, text is read as a number where it looks like one (`"3"`), `TRUE` is 1. * An omitted argument takes the function's default; a required argument that is left empty reads as 0. * A result that is not finite is `#NUM!`. Negative zero is written as 0. * Dates are serials truncated to whole days. A serial outside the calendar of the workbook's date system is `#NUM!`; text that is not a date is `#VALUE!`. ### Financial **Day-count bases** (`basis` 0 to 4, default 0; anything else `#NUM!`): 0 US (NASD) 30/360 with the last-day-of-February rules, 1 actual/actual, 2 actual/360, 3 actual/365, 4 European 30/360. On basis 1 a span inside one year uses 366 when it contains 29 February, and a longer span the average year length of the years it touches (`YEARFRAC`'s rule). The helper is `DayCount` (shared by every function that counts days). **Coupon dates** are counted back from maturity in steps of 12/`frequency` months (frequency 1, 2 or 4, otherwise `#NUM!`). A maturity on the last day of a month gives month-end coupon dates; otherwise the maturity's day of month is kept, clamped to the month's length. `settlement >= maturity` is `#NUM!`. `COUPDAYS` is the actual period on basis 1, 365/frequency on basis 3, otherwise 360/frequency; `COUPDAYBS` is the days from the previous coupon to settlement on the basis's count; `COUPDAYSNC` is the days from settlement to the next coupon (on bases 0 and 4 it is `COUPDAYS` minus `COUPDAYBS`). *(open: bases 0 and 4 for `COUPDAYSNC`)* **Time value.** `FV PV PMT NPER RATE IPMT PPMT ISPMT CUMIPMT CUMPRINC EFFECT NOMINAL RRI PDURATION FVSCHEDULE` follow the annuity equation `pv(1+r)^n + pmt(1+r·type)((1+r)^n−1)/r + fv = 0`; any non-zero `type` is 1 except in `CUMIPMT`/`CUMPRINC`, where it must be 0 or 1. `IPMT` with payments at the start of the period is 0 in period 1 and the end-of-period figure of the previous period, discounted one period, after that. `CUMIPMT`/`CUMPRINC` truncate `nper`, `start` and `end`, and require rate, nper and pv positive and `1 ≤ start ≤ end ≤ nper`. **Iteration.** `RATE`, `IRR` and `XIRR` use Newton's method from the guess (default 0.1). Excel documents 20 steps to 0.0000001 (100 for `XIRR`); plain Newton from 0.1 does not converge for long annuities that Excel solves (`RATE(360,-1000,150000)`), so ours halves a step until it keeps the rate above −1 and does not increase the residual, runs up to 100 steps and stops below 1e-12. `#NUM!` when it does not converge, when `IRR`/`XIRR` have no cash flow of each sign, or the guess is −1 or less. `IRR`/`MIRR` read only the numbers of a reference or array (text, booleans and blanks are skipped); `XNPV`/`XIRR`/`FVSCHEDULE` require every cell to be a number (a blank is 0, text is `#VALUE!`). *(open: iteration limit; `XIRR` of a zero result is a rounding-sized number here, Excel returns about 2.98E-9)* **Net present value.** `NPV` takes the numbers of references and arrays, and coerces direct arguments; `rate = -1` is `#DIV/0!`. `XNPV` uses a 365-day year from the first date; dates before the first are `#NUM!`; counts that differ are `#NUM!`. `MIRR` needs at least two values with both signs (`#DIV/0!` otherwise). **Depreciation.** `SLN SYD DB DDB VDB AMORLINC AMORDEGRC`. `DB` rounds the rate to three decimals and pro-rates the first and the (life+1)th period by `month`. `DDB` is declining balance that never goes below salvage; a fractional period interpolates linearly between whole periods. *(open)* `VDB` switches to straight-line over the remaining life when that is larger (unless `no_switch`), weights partial periods by their overlap, and limits `end` to the life. `AMORLINC`/`AMORDEGRC` implement the French system: a pro-rated first period, `AMORDEGRC` with the coefficient 1, 1.5, 2 or 2.5 by life (under 3, under 5, up to 6, over 6 years) and every charge rounded to a whole unit; basis 2 is `#NUM!`. *(open: the exhausted-asset tail of `AMORDEGRC`; `DB` with cost 0, which returns 0; fractional `life` and `period`)* **Securities.** `PRICE YIELD DURATION MDURATION` use the documented formulas with `A`, `E`, `DSC` from the coupon schedule; one coupon left uses the short formula. `YIELD` solves `PRICE` for the yield by bisection (the price falls monotonically as the yield rises, so the root is unique). `DISC INTRATE RECEIVED PRICEDISC YIELDDISC PRICEMAT YIELDMAT ACCRINTM` use the year fraction of the basis. `ACCRINT` sums the share of each quasi-coupon period (counted from `first_interest` in both directions) that lies between issue, or `first_interest` when `calc_method` is FALSE, and settlement. `ODDFPRICE`/`ODDLPRICE` use the quasi-coupon fractions for the odd period, so a long odd period works; `ODDFYIELD` solves by bisection and `ODDLYIELD` has a closed form. `TBILLPRICE/YIELD/EQ` need a bill of at most 365 days; `TBILLEQ` over half a year solves the quadratic with 366 days when the bill spans 29 February. *(open)* `DOLLARDE`/`DOLLARFR` read the decimals as a count of `fraction`ths with as many digits as `fraction` has. ### Engineering * **Base conversion** `BIN2DEC … OCT2HEX`: input text or number of at most 10 digits (`#NUM!` when longer or not in the base), 10 digits with the top bit set are negative (two's complement of 10, 30 or 40 bits); results are upper case; a negative result is always 10 digits and ignores `places`; `places` from 1 to 10 left-pads with zeros and is `#NUM!` when too small. A boolean is `#VALUE!`; an empty cell is 0; the empty string is `#NUM!` *(open)*. `DEC2x` truncates and accepts −512…511 (binary), −2^29…2^29−1 (octal), −2^39…2^39−1 (hex). * **Bits.** `BITAND BITOR BITXOR BITLSHIFT BITRSHIFT`: whole numbers from 0 to 2^48−1 (otherwise `#NUM!`); shifts of at most 53 places; a negative shift reverses direction; a left shift past 2^48 is `#NUM!`. * `DELTA`, `GESTEP`: numbers; omitted second argument 0. * **`ERF`/`ERFC`** (and `.PRECISE`): the series 2/√π·e^(−x²)·Σ 2ⁿx^(2n+1)/(2n+1)!! for |x| < 2.5, a continued fraction for the complement beyond; accurate to about 1e-15 relative. `ERF(lower, upper)` is `erf(upper) − erf(lower)`; negative limits are accepted. * **Bessel.** Order truncated; negative order `#NUM!`; `K` and `Y` need x > 0; |x| up to 1e6. `I` by its power series; `J` by the series where it does not cancel, otherwise the trapezoid rule on the periodic integral; `K` by the trapezoid rule on `∫e^(−x cosh t) cosh(nt)dt`; `Y` by Gauss–Legendre on its integral representation. Values are the exact ones to about 1e-15 (absolute near zeros). *(open: Excel's own Bessel values are less accurate: the reference prints `BESSELJ(1.9,2)` as 0.329925829 while the exact value is 0.3299257277.)* * **`CONVERT`**: the unit table of the Excel reference (mass, distance, time, pressure, force, energy, power, magnetism, temperature, volume, area, information, speed), case-sensitive; metric prefixes `Y Z E P T G M k h da/e d c m u n p f a z y` on the units marked metric, `ki Mi Gi Ti Pi Ei Zi Yi` on `bit` and `byte`; prefixes on area and volume units are raised to the unit's power (`km2` is (1000 m)²). Unknown units or units of different kinds are `#N/A`. Temperature converts through kelvins. *(open: the constants of a few rarely used units; `Schooner` and `Humpen`.)* * **Complex numbers.** Text `a+bi`, `a-bj`, `bi`, `i`, `-j`, or a plain number; no spaces; suffix `i` or `j` (lower case); a different suffix in two arguments is `#VALUE!`; text that is not a complex number is `#NUM!`. Results are written with 15 significant digits, a coefficient of 1 omitted (`3+i`, `-i`), and the suffix of the arguments (`i` by default). The trigonometric functions use the standard identities, `IMSQRT`/`IMPOWER` De Moivre's formula (so tiny rounding terms appear as in Excel), `IMLN(0)`, division by zero and the reciprocal functions at a pole are `#NUM!`, `IMARGUMENT(0)` is `#DIV/0!`. `IMSUM` and `IMPRODUCT` take ranges; blank cells in a range are skipped. ### Database `DSUM DAVERAGE DCOUNT DCOUNTA DGET DMAX DMIN DPRODUCT DSTDEV DSTDEVP DVAR DVARP (database, field, criteria)`. * **Database** and **criteria** are references or arrays; the first row holds the field names. (Excel accepts references only; accepting arrays lets a template evaluate the functions over data it holds as arrays. *(open)*) * **Field** is a 1-based column number or a header text compared ignoring case; anything else is `#VALUE!`. * **Criteria** header cells name database fields. Cells in one row must all match; the rows are alternatives. An empty cell places no condition and a row of empty cells matches every record; a criteria block with no data rows matches every record. A header that is not a database field makes its cells *computed* criteria: the record matches when the cell holds TRUE. * **A criterion** is matched as `COUNTIF` does (operators `= <> < <= > >=`, wildcards `? * ~`), except that text without an operator matches the cells that *begin* with it (`Dav` finds `Davolio`); `=Dav` is exact. * **Results.** `DSUM` adds the numbers of the field in the matching records (text and blanks are ignored; an error cell in a matching record is the result); `DAVERAGE` of no number is `#DIV/0!`; `DCOUNT` counts numbers, `DCOUNTA` non-blank cells; `DMAX`, `DMIN`, `DPRODUCT` of no number are 0; `DGET` returns the one match (`#VALUE!` for none, `#NUM!` for several); the sample deviation and variance of fewer than two numbers, and the population ones of none, are `#DIV/0!`. ### Grids `testdata/calc/function-grids/*.fgrid` hold typical, boundary, coercion, error and array inputs for every function (generated by `tools/calc-grids/gen_function_grids.py`). Excel 16.0.20527 answered them (`*.fgrid.tsv`; `mismatches.txt` lists the disagreements). The database grid reads the block described by `database.cells.tsv`, which the runner fills first. ### Open: financial, engineering and database, pending the 1.0 Excel grid 1. `COUPDAYSNC` on bases 0 and 4 (taken as `COUPDAYS` minus `COUPDAYBS`). 2. `XIRR` with a zero result returns a rounding-sized number here; Excel returns about 2.98E-9. The iteration limit of `RATE`, `IRR` and `XIRR`. 3. `DB` with cost 0; fractional `life` and `period` in `DB`, `DDB` and `VDB`; the exhausted-asset tail and coefficient boundaries of `AMORDEGRC`; basis 2 in `AMORLINC`/`AMORDEGRC` (`#NUM!` here). 4. `TBILLEQ` over half a year (366 days when the bill spans 29 February). 5. Base conversion of the empty string (`#NUM!` here) and of a blank cell (0). 6. A few `CONVERT` constants (`Schooner`, `Humpen`, `Pica`/`pica`, `parsec`) and which units take metric prefixes. 7. `IMSUM`/`IMPRODUCT` skipping blank cells in ranges; `IRR(TRUE)` is `#NUM!`. 8. The database functions accept arrays as well as references (Excel takes references only). ## Statistical Source: Microsoft's per-function reference pages ("Statistical functions (reference)" and each function's page). Numerics are in `Functions/Statistical/SpecialMath.cs`: log-gamma (Lanczos below 15, Stirling above), the error function (a series below 2.5, a continued fraction above), the regularized incomplete gamma and beta functions (series and continued fractions, each returning the tail that does not cancel), the normal quantile (a rational start refined by Halley steps) and a bisection solver for the other quantiles. `Exp`, `Ln`, `Pow`, `Log1p`, `Expm1` and `Tanh` come from OneMath so results are the same on every platform. **Reading the arguments.** `Stat.Gather`: in a reference or array only numbers count (text, logicals and blanks are skipped); a value typed in the argument list is converted (`"3"` and `TRUE` count, `"x"` is `#VALUE!`); an error anywhere is returned. The "A" forms (`AVERAGEA`, `MAXA`, `MINA`, `VARA`, `VARPA`, `STDEVA`, `STDEVPA`) count text in cells as 0 and `TRUE` as 1. `COUNT` never returns an error. Paired functions (`CORREL`, `SLOPE`, `COVARIANCE.*`, `FORECAST`, ...) drop a pair when either cell is not a number, return an error found in either, and give `#N/A` for arrays of different shapes. | Functions | Status | Notes | |---|---|---| | `COUNT COUNTA COUNTBLANK COUNTIF COUNTIFS` | docs | `COUNTIF(S)` use `CalcCriteria`. A criterion given as an array runs once per element and returns an array. Ranges of different shapes in `COUNTIFS` are `#VALUE!`. | | `AVERAGE AVERAGEA AVERAGEIF AVERAGEIFS MAX MAXA MIN MINA MAXIFS MINIFS` | docs | Sums use `Evaluator.AddSnapped`. No numbers: `AVERAGE` is `#DIV/0!`, `MAX`/`MIN`/`MAXIFS`/`MINIFS` are 0. `AVERAGEIF`'s `average_range` is resized to the shape of `range` from its own top-left cell. A matching cell holding an error returns the error. | | `MEDIAN MODE MODE.SNGL MODE.MULT LARGE SMALL RANK RANK.EQ RANK.AVG` | docs | `MODE.MULT` returns a column in order of first appearance. `LARGE`/`SMALL` round a fractional `k` up *(pending)*. `RANK` ignores non-numbers in `ref`; a number not in `ref` is `#N/A`. | | `PERCENTILE(.INC/.EXC) QUARTILE(.INC/.EXC) PERCENTRANK(.INC/.EXC)` | docs | Interpolation as the reference defines (`k(n-1)` inclusive, `k(n+1)` exclusive). `PERCENTRANK` truncates (does not round) to `significance` digits after rounding to 15 digits; one value in `.INC` is `#N/A` *(pending)*. | | `VAR VAR.S VAR.P VARP VARA VARPA STDEV STDEV.S STDEV.P STDEVP STDEVA STDEVPA DEVSQ AVEDEV GEOMEAN HARMEAN TRIMMEAN SKEW SKEW.P KURT STANDARDIZE` | docs | Two-pass sums of squares. `STDEV.S` of one value is `#DIV/0!`; `VAR.P` of one value is 0. `GEOMEAN`/`HARMEAN` of a value ≤ 0 are `#NUM!`. `TRIMMEAN` removes `floor(n*p/2)` from each end. | | `FREQUENCY PROB` | docs | `FREQUENCY` counts values in (previous bin, bin] and returns `bins+1` rows; unsorted bins are counted in ascending order and the counts put back in the order given *(pending)*. `PROB` needs every probability in (0,1] and a total of 1 within 1e-9. | | `CORREL PEARSON COVAR COVARIANCE.P COVARIANCE.S RSQ SLOPE INTERCEPT STEYX FORECAST FORECAST.LINEAR` | docs | See above for pairing. Zero variance or fewer than two pairs: `#DIV/0!` (`STEYX` needs three). | | `LINEST LOGEST TREND GROWTH` | docs | Householder QR. A predictor that is a combination of earlier ones is dropped: its coefficient and standard error are 0, as `LINEST` reports. `known_x` omitted is 1..n. `const` FALSE forces the intercept to 0 and uses the uncentred total sum of squares. With zero residual degrees of freedom the statistics rows give `#NUM!` for standard errors, `sey` and `F` *(pending)*. | | `FORECAST.ETS FORECAST.ETS.CONFINT FORECAST.ETS.SEASONALITY FORECAST.ETS.STAT` | gap | Microsoft's pages name the "AAA" exponential triple smoothing but do not give the algorithm (initial states, parameter fit, seasonality detection), so all four return `#N/A`. Listed as a gap; not guessed. | | `NORM.DIST NORM.S.DIST NORM.INV NORM.S.INV LOGNORM.DIST LOGNORM.INV PHI GAUSS FISHER FISHERINV` and legacy `NORMDIST NORMSDIST NORMINV NORMSINV LOGNORMDIST LOGINV` | docs | Legacy names share the implementation (`LOGNORMDIST`, `NORMSDIST` are cumulative only). | | `T.DIST T.DIST.2T T.DIST.RT T.INV T.INV.2T CHISQ.DIST CHISQ.DIST.RT CHISQ.INV CHISQ.INV.RT F.DIST F.DIST.RT F.INV F.INV.RT BETA.DIST BETA.INV GAMMA.DIST GAMMA.INV` and legacy `TDIST TINV CHIDIST CHIINV FDIST FINV BETADIST BETAINV GAMMADIST GAMMAINV` | docs | Degrees of freedom are truncated; below 1 or from 1e10 is `#NUM!`. Quantiles are solved to the last bit of the double, so they do not reproduce Excel's iteration limit (`#N/A` from `BETA.INV` after 100 steps) *(deviation)*. | | `BINOM.DIST BINOM.DIST.RANGE BINOM.INV POISSON.DIST NEGBINOM.DIST HYPGEOM.DIST EXPON.DIST WEIBULL.DIST GAMMA GAMMALN GAMMALN.PRECISE` and legacy `BINOMDIST CRITBINOM POISSON NEGBINOMDIST HYPGEOMDIST EXPONDIST WEIBULL` | docs | Counts are truncated after the sign check. Cumulative sums are exact term sums for small sizes and the incomplete beta/gamma functions beyond. More than 1e15 trials, or a cumulative hypergeometric sum of more than 5e6 terms, is `#NUM!` *(deviation: Excel's limits are not documented)*. | | `CONFIDENCE CONFIDENCE.NORM CONFIDENCE.T` | docs | `CONFIDENCE.T` with size 1 is `#DIV/0!`. | | `Z.TEST ZTEST T.TEST TTEST F.TEST FTEST CHISQ.TEST CHITEST` | docs | `T.TEST` type 3 uses the Welch degrees of freedom unrounded. `F.TEST` is the two-tailed probability. `CHISQ.TEST` degrees of freedom are `(r-1)(c-1)`, or `max(r,c)-1` for one row or column. | ## Date and time Source: the per-function pages under "Date and time functions (reference)". Serials are counted in `EvaluatorOptions.DateSystem` (1900 with its leap-year bug, or 1904); all calendar arithmetic goes through `ExcelDate`. `TODAY` and `NOW` are volatile and read the local time of `ICalcResolver.Clock`, else `EvaluatorOptions.Clock`, else the system clock (the only place the evaluator reads the wall clock). Date arguments may be serials, logicals or text; text is read by `DateText` (below) before falling back to a number. | Functions | Status | Notes | |---|---|---| | `DATE TIME` | docs | `DATE`: a year of 0-1899 has 1900 added, 10000 and over is `#NUM!`, months and days roll over (`DATE(2008,14,2)` is 2009-02-02, `DATE(1900,2,29)` is 60, `DATE(1900,1,0)` is 0), a negative serial is `#NUM!`. `TIME`: each part is truncated, parts above 32767 are `#NUM!`, the day part is dropped. Negative parts are `#NUM!` *(pending)*. | | `DATEVALUE TIMEVALUE` | docs | `DateText`: `m/d/y`, `y-m-d`, `d-mmm-y`, `mmm d, y`, `d mmm y`, month-year, month-day and numeric month/day (current year from the clock); two-digit years pivot at 30; times with 24-hour or AM/PM clocks and fractional seconds; day names are ignored. English names and US order only. A time alone is a `#VALUE!` for `DATEVALUE` and a date alone is 0 for `TIMEVALUE` *(pending)*. | | `YEAR MONTH DAY HOUR MINUTE SECOND` | docs | Negative serials and serials past 9999-12-31 are `#NUM!`. Times are rounded to the millisecond first (`ExcelDate`), so `SECOND(0.99999999)` is 0 *(pending: Excel may round to the second)*. | | `WEEKDAY WEEKNUM ISOWEEKNUM` | docs | Types 1, 2, 3, 11-17 (`WEEKDAY`); 1, 2, 11-17, 21 (`WEEKNUM`). Serial 1 of the 1900 system is a Sunday as in Excel; serial 0 is a Saturday and `WEEKNUM(0)` is 0. | | `EDATE EOMONTH` | docs | The day is clamped to the end of the target month. `EOMONTH` into February 1900 gives 60. | | `DAYS DAYS360 DATEDIF YEARFRAC` | docs | `DAYS360` US: a start on the last day of a month counts as the 30th, an end on the 31st counts as the 30th once the start is the 30th; both on the last day of February make the end the 30th. `YEARFRAC` basis 1: the span within one year uses 366 when it holds a 29 February, longer spans use the average length of the years touched. `DATEDIF` units `Y M D MD YM YD`; `MD` borrows the length of the month before the end date's month. The 30/360 end-of-February rule and the `DATEDIF` `MD` borrow are *(pending)*. | | `NETWORKDAYS NETWORKDAYS.INTL WORKDAY WORKDAY.INTL` | docs | Weekend numbers 1-7 and 11-17, or a seven-character Monday-first string (`1111111` is `#VALUE!`; a wrong length is `#VALUE!`; an unknown number is `#NUM!`). Holidays may be any array or range of dates, blanks are skipped, text that is not a date is `#VALUE!`. `WORKDAY` with more than 4,000,000 days is `#NUM!`. | ## Open: statistical and date/time, pending the 1.0 Excel grid The generated grids are `testdata/calc/function-grid/statistical.*` and `datetime.*` (`tools/function-grid-stat/gen.py`); answers are recorded at the 1.0 gate. 1. `LARGE`/`SMALL` with a fractional `k` (rounded up here); `PERCENTILE` with a fractional `k` position is interpolated. 2. `PERCENTRANK` with a single value, and the exact rounding of the truncation to `significance` digits. 3. Average/sum rounding in `AVERAGE` for sums that cancel (the same snap as `SUM` is used). 4. `FREQUENCY` with unsorted bins, blanks or text in `bins_array`. 5. `LINEST` statistics with zero residual degrees of freedom, with `const` FALSE, and which of several collinear columns Excel drops. 6. `TTEST`/`T.TEST` with arrays of different sizes for type 1 (`#N/A` here) and pairs holding text. 7. The accuracy bound of the distribution kernels (about 1e-14 relative in the bulk and about 1e-13 for `GAMMA` of non-integers above 100), and which results Excel returns as `#NUM!` for extreme arguments (huge degrees of freedom, `p` within 1e-300 of 0 or 1). 8. `TIME` with negative parts, `DATEVALUE` of a time alone, and `TIMEVALUE` of a date alone. 9. Rounding of `HOUR`/`MINUTE`/`SECOND` for fractions within a millisecond of the next second. 10. `DAYS360` end-of-February rules, `YEARFRAC` basis 1 across several years, `DATEDIF` `MD` across a short month. 11. `EDATE`/`EOMONTH` from serial 0 and 60 (1900-01-00 and the fictitious 1900-02-29); `WEEKNUM`/`ISOWEEKNUM` before 1900-03-01. 12. Date text: `"1/5"` (current year), `"Jan-20"` (day or year), `"2020-01"`, 2-digit years at the pivot, `"Mon 1/1/2020"`, and text with both a date and a time given to `YEAR` etc. 13. `WEEKDAY(0)` and 1904-system weekdays (computed from the calendar, since 1904-01-01 is a Friday). # Number formats and date serials Normative notes for `RecursiveAI.OneDocument.NumberFormats`: Excel's number-format code language as one shared service, and the date-serial rules it and the workbook need. The calc engine's `TEXT` and `VALUE`, cell rendering, text export and templates all call it, so a value reads the same everywhere. Sources: ECMA-376 Part 1 §18.8.30 (`numFmt`, the built-in IDs and the General rules), §18.8.31 (`numFmts`), §18.2.28 (`date1904`); Microsoft's "Number format codes" and "Review guidelines for customizing a number format" articles; the `CELL` function reference. Excel is the specification for everything these pages leave open. Rules marked *observed in Excel* were read back from desktop Excel (365, build 16.0.20527, US English settings) over a grid of 7,602 code and value pairs (see "Verification"); the answers are committed in `testdata/numfmt/excel-grid.tsv` and checked by a test, so the formatter agrees with Excel on every case of the grid. The few things the grid did not reach are listed under "Not verified". ## API | Type | What it is | |---|---| | `NumberFormat` | A parsed, immutable, thread-safe format code. `Parse` (throws `OfficeFormatException`, defect `numfmt.*`), `TryParse` (returns a `NumberFormatDefect`), `Format(double)`, `TryFormat(double, Span)` (allocation-free), `FormatText(string)`, `TryFormatText`, `Category`, `Sections`, `GetCellFormatCode()`, `GetBuiltInCode(id, culture)`, `FindBuiltInId(code)`. | | `FormatOptions` | Date system, culture, General width (default 11), maximum width (default none). | | `FormatResult` / `FormatStatus` | Status, length, section colour and the `*` fill hint. | | `NumberFormatCulture` | Separators, month and day names, AM/PM, currency and date patterns for 24 locales, compiled in. | | `ExcelDate` | Serial ↔ date and time for both date systems. | Cell values stay numbers; a date is a serial. Formatting never reads the host culture: the output is the same with ICU, in invariant-globalization mode, under NativeAOT and in the browser. ## Sections and selection A code has up to four sections separated by `;` outside quotes, brackets and escapes: positive, negative, zero, text. An empty code is `General`. * One section applies to every number. Two: the first is for positive numbers and zero, the second for negatives. Three or four: positive, negative, zero (the fourth is text). * A condition (`[>100]`, `=`, `<>`, `<`, `>`, `<=`, `>=`) replaces the default test of its section. At most the first two numeric sections may need one; the last numeric section takes whatever is left. If nothing matches, the last numeric section is used. * The minus sign is shown before everything else in the section (`-$5`, `-x1`). It is left out when the section is for negative numbers only. *Observed in Excel*: * with no condition anywhere, the second and third sections are the negative ones: `0;0` shows `5` for -5; * a section with a condition drops the sign when the condition holds only below zero (`[<0]`, `[<-3]`, `[<=-3]`, `[=-5]`); `[<=0]`, `[<5]` and `[<>-5]` keep it (`[<=0]"n";0` shows `-n`); * a leftover section after conditions drops it when the first condition is `<`, `<=` or `<>` (`[<=-3]0;0.0` shows `1.0` for -1), and keeps it after `>`, `>=` or `=` (`[>100]"big";0.0` shows `-5.0`). * *Observed in Excel*: a negative number that rounds to zero shows no minus sign (`0.00` shows `0.00` for -0.001), except in a fraction with a whole part (`# ?/?` shows `-0` plus spaces). A negative zero is zero. * *Observed in Excel*: when every section has a condition and none matches (`[>0]0;[<0]0` for 0), nothing can be shown: `FormatStatus.NotRepresentable`, a row of `#`. A section that holds only a colour or a condition (`[Red][<=100];[Blue][>100]`) shows the number as General. * Text: the fourth section, or else the first section that contains `@` and no digit placeholder. A format without a text section shows text unchanged. A text section without `@` shows only its literals. *Observed in Excel*: a number put through a format that has only a text section (`@`, `"x"@`) shows as General, without the literals; and `@` next to digit placeholders (`0.00@`) is an invalid code. * `[Red]`, `[Color1]` to `[Color56]` and the other seven names set the section colour (reported, never applied). `[DBNum1-3]` rewrites digits as CJK numerals. ## Literals and layout hints `"text"`, `\c`, and the characters Excel allows unquoted (`$ - + / ( ) : ! ^ & ' ~ { } < > =` and space) are literal; so are non-ASCII characters. An unquoted ASCII letter that is not a code is the defect `numfmt.unexpected-character`. `_c` writes one space. `*c` writes nothing and reports `FillPosition`/`FillCharacter` so the renderer can repeat the character to the cell width. `[$text-LCID]` writes `text` (a currency tag) and selects the locale for names and calendar; `[$-F800]` and `[$-F400]` are the system long date and time (resolved against US English at parse time). ## Numbers Excel shows at most 15 significant digits. A double is first reduced to 15 significant decimal digits (correctly rounded from the exact binary value), and everything after (percent and thousands scaling, rounding, digit placement) works on those decimal digits with round-half-away-from-zero. So `0.285` with `0.00` is `0.29`, `1.005` is `1.01`, `0.1+0.2` is `0.30`, and `1234567890123456789` with `0` is `1234567890123460000`. * `0` always shows a digit, `#` only a significant one, `?` a space in place of an insignificant zero. Extra integer digits are always shown. With only `#` left of the point, a value below 1 starts with the point (`.47`); the point itself is always written when the code has one (`0.#` shows `1.`). * A comma between placeholders groups thousands (the culture's separator); after the last placeholder each comma divides by 1000. `%` multiplies by 100 and is written where it stands (each `%` counts). * Scientific: `E+`/`E-` followed by 0/# for the exponent digits. *Observed in Excel*: only the capital E; `0.00e+00` is invalid (`numfmt.exponent-without-digits`). The integer placeholder count sets the exponent step (`##0.0E+0` is engineering). Rounding that carries into a new digit moves the exponent. For the value zero every integer placeholder shows `0` (`#0.0E+0` gives `00.0E+0`). * Fractions: `# ?/?`, `# ??/??`, `?/?` (improper), fixed denominators (`# ?/8`). *Observed in Excel*: the fraction is the last continued-fraction convergent whose denominator fits the placeholders and whose numerator fits 31 bits, computed on the binary double (so 5.3 gives `5 2/7` and 0.1+0.2 gives `1/3`), not the closest fraction; when not even the first step fits, the number is rounded to a whole (`0.08` with `?/?` is `0/1`). A numerator past 2147483647 cannot be shown: `FormatStatus.NotRepresentable`. A fixed denominator rounds `fraction x denominator` to 15 digits first, then half away (`0.285` over 100 is `29/100`). A zero numerator after a whole part shows blanks for the whole fraction (`2` with `# ?/8` is `2` plus four spaces), unless the numerator has a `0` placeholder (`0 0/0` shows `2 0/1`). Without a whole part the fraction keeps its zero (`0/1`) and no minus sign. * `General`: up to `GeneralWidth` characters (11; the sign is not counted), at most 15 significant digits. Scientific (`1.23457E+11`) when the integer part is wider than that. *Observed in Excel* (`TEXT` and cells alike): below one, the plain form wins unless the scientific form keeps more of the digits; `0.00001` and `0.000000001` stay plain, `0.000123457` stays plain, `1.2345E-08` and `1.2E-10` go scientific. `GeneralWidth` is a parameter. A caller such as `TEXT` that refuses results over 255 characters does so itself: `1E+300` with `0` is 301 characters, which `TEXT` reports as `#VALUE!`, and a cell shows a row of `#` when the text does not fit its column. * `MaxWidth`: when the caller gives a width and the text is wider, `TryFormat` returns `DoesNotFit` and a row of `#` of that width (not for General, which fits itself). ## Dates and times Codes: `y yy yyy yyyy`, `m mm mmm mmmm mmmmm`, `d dd ddd dddd`, `h hh`, `m mm` (minutes), `s ss`, `AM/PM am/pm A/P a/p`, `[h] [m] [s]` (elapsed, `[hh]` pads), `.0 .00 .000` after seconds, `e ee`, `g gg ggg`, `b bb bbbb`. `m`/`mm` are minutes right after an hour code or right before a seconds code (literals between are skipped), otherwise months. AM/PM selects the 12-hour clock. `y` and `yy` are two digits, `yyy` and `yyyy` four. A date or time section that has digit placeholders is the defect `numfmt.mixed-codes`. * Serials: 1900 system: 0 is "1900-01-00" (a Saturday), 1 is Sunday 1900-01-01, 60 is the fictitious 1900-02-29, 61 is 1900-03-01 and later serials are real days (1899-12-30 plus the serial). 1904 system: 0 is Friday 1904-01-01. The largest serials are 2958465 and 2957003 (9999-12-31). Weekdays follow Excel's count, so serials below 61 differ from the real calendar. * Times are fractions of a day rounded to the nearest millisecond; for display they are rounded to the precision of the code (whole seconds unless `.0` digits are present), and a time that rounds up to a whole day carries into the next date (23:59:59.9996 shows `00:00:00` the next day). * A date or time format of a negative number, or one past the last day, is not shown: `FormatStatus.DateOutOfRange`; `Format` returns `#` characters. *Observed in Excel*: a format whose only date or time part is one elapsed field (`[h]`, `[m]`, `[ss]`) is a plain count: no range limit, a minus sign for negative values, rounded to whole seconds first and then cut to the unit (0.99999999 days is `24` hours); `[h]:mm` and the like keep the range. * *Observed in Excel*: `am/pm` in lower case still shows `AM` and `PM`; only `a/p` shows lower case. The written pairs `上午/下午`, `午前/午後` and `오전/오후` are AM/PM codes that show their own half. `[$-F800]` is `dddd, mmmm d, yyyy` (no padded day). * `e`, `g`: Gregorian year unless the locale tag or its calendar bits say otherwise: Japanese eras (Meiji from 1868-09-08, Taisho, Showa, Heisei, Reiwa; `g` one letter, `gg` the first kanji, `ggg` the era name), Minguo (a tag whose calendar byte is 4, year-1911, `民國`). *Observed in Excel*: `[$-404]e` alone is the Gregorian year, and the Gregorian calendar has no era name (`g` is empty). `b` is the Buddhist year on its own, but the Gregorian year when a locale tag without the Thai calendar is present (`[$-41E]bbbb` is 1900 for serial 1). Locale tags `[$-xxxxxxxx]`: the low 16 bits are the LCID; the next byte is a Windows calendar ID (3 Japanese, 4 Taiwan, 7 Thai); the byte above that is the numeral system (`0D` is Thai digits). With calendar 7, `yyyy` is the Buddhist year; other calendars (Hijri, Hebrew) fall back to Gregorian. ## Numerals `[DBNum1]`, `[DBNum2]` and `[DBNum3]` write digits as CJK numerals, but only with a Chinese, Japanese or Korean locale (in the tag, or `FormatOptions.Culture`); without one they change nothing, as in Excel on an English system. *Observed in Excel* (zh-CN, zh-TW, ja-JP, ko-KR): * The digits of a number are written one by one: 123 is `一二三`, 105 is `一○五` (zero is `○` U+25CB for zh, `〇` U+3007 for ja, `０` for ko), including digits after the point and grouping separators in place (`一,○○○`). * A year, a padded part (`mm` of 05) and any digit after a point are digit by digit too. An unpadded month, day, hour, minute or second is a place-value numeral: `十二月三十一日`; DBNum2 `壹拾贰月叁拾壹日`. * DBNum2 digits: zh-CN `零壹贰叁肆伍陆柒捌玖`, zh-TW `零壹貳參肆伍陸柒捌玖`, ja-JP `〇壱弐参四伍六七八九`, ko-KR like zh-TW except that zero is U+F9B2, three U+F96B and four `四`. DBNum3 is full-width digits `０-９` everywhere. * Strings longer than 256 characters keep plain digits. ## Cultures `NumberFormatCulture` covers en-US, en-GB, de-DE, fr-FR, es-ES, it-IT, pt-BR, nl-NL, ru-RU, ja-JP, zh-CN, zh-TW, ko-KR, th-TH, hu-HU, da-DK, nb-NO, sv-SE, fi-FI, lt-LT, pl-PL, cs-CZ, tr-TR and dv-MV (the last eleven were added after workbooks in the wild used them; their names, weekday and AM/PM texts are Excel's own, 25 tags and 1,350 cells in `testdata/numfmt/excel-locales.tsv`). Under any `[$-LCID]` tag, `A/P` and `a/p` show the locale's whole AM/PM text (observed in Excel, even for `[$-409]`); untagged they are one letter. It holds the decimal and group separators (French groups with U+202F, Russian with U+00A0), month and day names, AM/PM text, the local currency symbol and the short and long date patterns. The names were compared with Excel's output for every month and weekday (`mmmm`, `mmm`, `dddd`, `ddd`). Where Excel differs from the usual convention, Excel wins: Russian month and short weekday names are capitalised and nominative even next to a day (`15 Март 2023`), Spanish short names end in a full stop (`ene.`, `ma.`), Japanese and Korean `mmm` are the bare number, zh-TW `mmm` is the long name. A `[$-LCID]` tag changes names, AM/PM and calendar for its section; separators always come from `FormatOptions.Culture`. A locale outside the table falls back to its language, then to US English. Built-in IDs 0 to 49 come from `NumberFormat.GetBuiltInCode`. IDs 5 to 8, 14, 22 and 41 to 44 follow the culture's currency symbol and date pattern; 37 to 40 are written as Excel saves them (`#,##0_);(#,##0)`; the ISO text shows a space where Excel writes `_)`). IDs 27 to 36 and 50 to 58 are defined for zh-TW, zh-CN, ja-JP and ko-KR, and are `null` for other cultures. IDs 59 to 81 (Thai) are not implemented. ## Not verified The grid did not reach: Korean `DBNum2` digits 6 to 9 (written with the traditional forms, not read back); Hijri and Hebrew calendars (fall back to Gregorian); `General` for widths other than 11; locales outside the table; right-to-left and Thai built-in IDs 59 to 81. ## Categories and `CELL("format")` `Category`: General, Number, Currency, Accounting (a `*` fill), Date, Time, DateTime, Percentage, Fraction, Scientific, Text, Custom, from the first section that applies to numbers. `GetCellFormatCode()` returns the `CELL` codes `G`, `F0`, `,2`, `C0`, `P2`, `S2`, `D1`…`D9`, with `-` when negative numbers are coloured and `()` for parentheses; codes Excel has no letter for give `G`. ## Defects Invalid codes are never exceptions of another type. `TryParse` returns a `NumberFormatDefect` (code, message, position in the code); `Parse` throws `OfficeFormatException` with the same code. Codes: see `docs/defects.md` (`numfmt.*`). ## Verification * `tools/NumFmtGrid` writes the edge-case grid: `grid-text.xlsx` (`=TEXT(value, code)`) and `grid-cells-NN.xlsx` (values in cells formatted with the code, in shards of 12 codes so that a code Excel refuses costs one shard), and our answer for each case in `manifest.tsv`. `tools/office-oracle` (`run --grid`) opens each workbook in desktop Excel and writes the answers back; `NumFmtGrid compare` reports agreement. `tools/NumFmtGrid/run-libreoffice.sh` does the same with LibreOffice headless (information only; its null date, truncated seconds and `?` spacing differ from Excel). * Agreement with Excel 16.0.20527 over the 7,602 cases: `TEXT()` 7,602 and cell display 7,602, after setting aside what is not the formatter's: `TEXT` turns numeric-looking text into a number, refuses results over 255 characters, and a cell shows `#` rows when the text is too wide and stretches `*` fills. The two codes Excel rejects (`0.00e+00`, `0.00@`) are rejected here. * `NumberFormatExcelGridTests` replays `testdata/numfmt/excel-grid.tsv` against the formatter. * Unit tests take their expected strings from Microsoft's articles (cited in the test file) or from the rules above. * Fuzz targets `numfmt` (parser and formatter on arbitrary codes) and `numfmtfmt` (string, span and exact-size span agree). * The span path allocates nothing; a test measures it over thousands of calls. # CSV and TSV Delimited text is a native layer of `RecursiveAI.OneDocument.Spreadsheets` (namespace `RecursiveAI.OneDocument.Spreadsheets.Csv`): a streaming reader, a value parser, a streaming writer, and a bridge to the workbook model. The rules below are RFC 4180 (§2, definitions of the format) as the baseline, extended the way Excel reads and writes the files. Where Excel's behaviour was not documented it was found by opening generated files in Excel 16.0 (build 20527) and comparing cell by cell; the generator and the comparison are `tools/csv-grid/` and `tools/CsvProbe/`. ## Reader (`CsvReader`) Forward-only. After `ReadRecord()` the fields of the record are spans over the reader's buffer, already unquoted. Reading allocates nothing per field once the buffers have grown to the largest record, and memory is the largest record, not the file. One record is limited by `OfficeLimits.MaxAllocationBytes`. - **Records** end at CR LF, LF or a lone CR (Excel reads all three). A final line break does not start a record. An empty line is a record of one empty field (`IsBlank`), as Excel keeps it as an empty row; `SkipBlankLines` drops them. A delimiter at the very end of the input still gives a last empty field. - **Quoting.** A field that starts with the quote character is quoted: it may hold delimiters, line breaks and doubled quotes (`""` is one quote). A quote anywhere else in a field is ordinary text (`5" pipe`). Text between a closing quote and the next delimiter is kept after the quoted text (`"ab"cd` is `abcd`). - **Delimiter.** Set it, or leave it null to detect it: of comma, semicolon, tab and pipe, the one that splits each of the first (up to) 20 records into the same number (more than one) of fields outside quotes, from the first 32,768 characters; otherwise the one that occurs most; otherwise comma. A first line `sep=X` (what Excel writes and reads) names the delimiter and is not data. - **Encoding.** A byte-order mark wins: UTF-8, UTF-16 little endian, UTF-16 big endian. Without one: the caller's `Encoding` if given; else UTF-8 when the bytes are valid UTF-8 (a seekable stream is checked whole; a forward-only stream by its first block), else Windows-1252, which the reader decodes with its own table, so no code-page provider is needed (defect `csv.encoding-assumed`). Excel reads UTF-8 without a mark as the system's legacy code page and shows mojibake; the reader deliberately does better. Any other code page needs the caller to register `CodePagesEncodingProvider` and pass the `Encoding`, which is a platform decision (trimmed and WebAssembly hosts may not have the tables). - **Ragged records** (a different field count from the first record) are returned as they are, never padded or cut, with `csv.ragged-record` located by character offset. ### Defects | Code | Severity | Meaning | |---|---|---| | `csv.ragged-record` | repaired | A record has a different number of fields from the first. | | `csv.stray-quote` | repaired | Text follows the closing quote of a quoted field; it is kept after the quoted text. | | `csv.quote-unterminated` | repaired | A quoted field is open at the end of the input; it ends there. | | `csv.encoding-assumed` | repaired | No byte-order mark and not valid UTF-8; read as Windows-1252. | | `csv.encoding-invalid` | repaired | A byte sequence is not valid in the encoding (or UTF-16 ends in half a unit); it reads as U+FFFD (or is dropped). Reported once per reader. | | `csv.grid-exceeded` | dropped | Import only: more than 1,048,576 records or 16,384 fields in a record; the rest are dropped. | ## Values (`CsvValueParser`) Reading is separate from splitting: a field is text until the parser says otherwise, and `CsvValueOptions.Infer = false` keeps everything as text. The parser allocates nothing per call. Booleans and errors must be the whole field (Excel keeps `" TRUE"` as text); everything else ignores spaces around the field. What is recognised, with the number format Excel gives the cell (en-US): | Text | Value | Format | |---|---|---| | `42`, `-3.5`, `+7`, `.5`, `5.`, `00` | number | General | | `1,234`, `1,234.5`, `1,2345` | number | `#,##0`, `#,##0.00` | | groups: first group 1–3 digits without a leading zero, every later group 3 or more digits | | `1,23`, `12,34`, `0,123`, `1,234,56` stay text | | `--5`, `+-5` | up to two signs, each `-` flips | | | `$5`, `-$5`, `$-5`, `($5)`, `$1,234.50` | number | `$#,##0_);[Red]($#,##0)`, with `.00` when decimals were written | | `(5)`, `(1,234)` | negative number | General / `#,##0`; a sign inside the parentheses is text; `5-`, `5$`, `5+` are text | | `1E3`, `1.5e-2`, `1,000E3` | number | `0.00E+00` | | `5%`, `5 %`, `%5`, `(5%)`, `0.5%`, `1,000%` | fraction | `0%`, `0.00%` when decimals were written; `5%%` is text; `1E3%` is the number 10 | | `TRUE`, `false` (any case) | boolean | | | `#N/A`, `#DIV/0!`, `#VALUE!`, `#REF!`, `#NAME?`, `#NUM!`, `#NULL!`, `#SPILL!`, `#CALC!`, `#GETTING_DATA` | error | | | `1/2/2020`, `1-2-20`, `2020-01-02`, `2020/1/2` | date | `m/d/yyyy` (the culture's short date elsewhere) | | `2-Jan-2020`, `2 Jan 20`, `2 January 2020`, `Jan 2, 2020`, `Sept 2, 2020` | date | `d-mmm-yy` | | `Jan 2020`, `Jan-2020`, `January 2020` | first of the month | `mmm-yy` | | `13:45`, `1:45`, `1:45 PM`, `1:45 P`, `12:00 AM`, `0:30 AM`, `1:` | time of day | `h:mm`, `h:mm AM/PM`, `h:mm:ss` with seconds, `mm:ss.0` with fractions of a second | | `24:00`, `25:30`, `100:00` | elapsed time | `[h]:mm:ss` | | date and time joined by one space | date and time | `m/d/yyyy h:mm` | Text that Excel also keeps as text: `1/1/1899` and earlier (years before 1900), `2/29/2019`, `13/1/2020`, `2020-02-30`, `99-01-02` (two-digit year first), `20200102` is a *number*, `2020.01.02`, `Jan 2 2020` and `Jan-2-2020` and `2 Jan, 2020` (the comma is required in the month-first form and forbidden in the day-first form), `Monday, January 2, 2020`, `1:45PM` (a space is required before AM/PM), `13:45 AM`, `2020-01-02T13:45:30` (ISO 8601 with `T` is not read by Excel), `1.2.2020` (only where the culture's date separator is a dot), numbers outside Excel's range (below 2.2250738585072014E-308 or above 9.99999999999999E+307 in magnitude), `Infinity`, `NaN`, `0x1F`. ### Deliberate differences from Excel - **A date without a year** (`1/2`, `1-2`, `2-Jan`, `Jan 2`, `Jan-20` meaning 20 January) is a date in the current year in Excel. The library never reads the clock, so these stay text unless `CsvValueOptions.DefaultYear` gives the year; with it they read as Excel does (`d-mmm` format). - **Leading zeros and long numbers.** Excel turns `007` into 7 and keeps 15 significant digits of `12345678901234567890`. By default a whole number written with a leading zero, or with more than 15 digits, stays text (`PreserveLeadingZeros`, `PreserveLongNumbers`); set them false to match Excel, which then keeps 15 significant digits (later integer digits become zeros, later fraction digits are dropped). - **Formulas.** Excel evaluates a field such as `=1+1` (and `-Infinity`, as `=-Infinity`) when it opens the file. The reader keeps them as text: a CSV is data, and running it is the injection the writer defends against. - **Encoding** as above. - **Two-digit years** use the pivot Excel 16.0.20527 showed on its default settings (`TwoDigitYearPivot` = 50: 00–49 are 2000–2049, 50–99 are 1950–1999; checked with 1/2/29, 30, 31, 49, 50, 51, 99, 00, 1). ## Writer (`CsvWriter`, `CsvDocument.Write`) `CsvWriter` writes fields one at a time and ends records with `EndRecord()`; it buffers a little and holds no record, so any size is constant space. Defaults are Excel's: comma, double quote, CR LF, UTF-8 **with** a byte-order mark (without it Excel garbles non-ASCII text; `ByteOrderMark = false` for tools that dislike the mark), quotes only where needed (a field with the delimiter, a quote, CR or LF), an empty field as nothing. `Quoting = Always` quotes every non-empty field. The encoding is any `Encoding`; unmappable characters become `?` (the platform's replacement). `CsvDocument.Write(sheet | workbook, stream, options, loss)` writes the used range from A1, one record per row (empty rows are empty lines, empty cells are empty fields), the first visible worksheet of a workbook. A formula cell writes its cached value. With `Formatted = true` (the default, as Excel's Save As) a number is written as its number format displays it (`General` at up to 20 characters, so full precision is kept; dates as dates), with `Formatted = false` raw: the shortest text that reads back as the same double, dates as serial numbers. Booleans are `TRUE`/`FALSE`, errors their text. ### Formula injection A spreadsheet runs a field that starts with `=`, `+`, `-` or `@` (and, in some programs, a tab or carriage return). With `FormulaInjection = Escape` — **the default** — *text* that starts with one of these gets a single quote put in front of it and is written quoted: `"'=1+1"`. Numbers written as numbers (a negative amount) are never touched. The default is on because a CSV is most often opened in a spreadsheet and the cost is a visible quote on the few real texts that start with these characters; programs that read the file back choose `None`. `CsvWriter.WriteField(text, isText: false)` marks the text of a number. ### Loss report Writing a sheet reports into an `ILossSink` (a `LossReport`): `loss.formulas` (cells, written as cached values), `loss.cell-styles` (cells with a style other than the default), `loss.number-formats` (cells that had one, in raw mode only), `loss.sheets` (sheets not written), `loss.merged-cells` (ranges), `loss.embedded-objects` (drawings, charts, tables, comments, pivot tables). ## The workbook `Workbook.Open` finds delimited text by content (the sniffer wants two complete lines with the same number of tabs, commas or semicolons) and gives a workbook of one worksheet (`Sheet1`), with every field a cell: values typed as above, dates, times, percentages, grouped numbers and currency with a number format Excel would give them (a fixed style sheet is generated), text as strings. A tab-separated content is read with a tab. `CsvDocument.ToWorkbook(stream | path, CsvImportOptions)` sets the delimiter, encoding, inference, a header row (kept as text) and the sheet name. `workbook.SaveAs("x.csv")` and `SaveAs("x.tsv")` write the first visible sheet with the defaults. Cell counts are bounded by `OfficeLimits.MaxRecordCount`. ## What was checked Reading against Excel 16.0.20527: two value grids (262 and 193 cells) and 14 shape files opened in Excel and read cell by cell: 259/262 and 193/193 agree (the three differences are deliberate: Excel evaluates `=1+1` and reads `-Infinity` as a formula name; we keep text unless formulas are asked for). Files written by the writer, read back by Excel: 162/162 raw and 162/162 formatted cells equal, number formats included. Fuzzing: `fuzz/run.sh csv`. Corpus: 89 CSV files read with typed defects only, field-equal after a rewrite. # Binary workbooks (XLSB) `RecursiveAI.OneDocument.Xlsb` (in the spreadsheets package) reads and writes the Excel binary workbook ([MS-XLSB]). The format is an OPC package whose parts hold records instead of XML. `Workbook.Open` finds it from the content (the workbook part's content type), shows it through the same `Workbook`/`Worksheet`/`Cell` model as SpreadsheetML (`Workbook.IsBinary`), and `SaveAs("x.xlsb")` or `SaveAs(stream, OfficeFormat.Xlsb, loss)` writes it. `XlsbStreamReader` is the forward-only reader. ## Layers 1. **Records** (`XlsbRecordReader`, `XlsbRecordWriter`, `XlsbRecordData`, `XlsbRecordFields`, `XlsbRecordType`). The framing of [MS-XLSB] §2.1.4: a type of one or two bytes, a size of one to four. A record whose declared size passes the end of the part is a `xlsb.record-truncated` defect and ends the part; a record larger than `OfficeLimits.MaxPartSize` raises a limit exception before anything is allocated. Types and sizes are range-checked on write. 2. **Part translators** (`XlsbWorkbookPart`, `XlsbSheetPart`, `XlsbStylesPart`, `XlsbSharedStringsPart`, `XlsbCommentsPart`, `XlsbTablePart`, `XlsbChartSheetPart`) turn one binary part into SpreadsheetML and back. They are lazy: a part is translated the first time the hub asks for it. 3. **Formulas** (`XlsbFormula`, `XlsbFormulaContext`, `XlsbFunctionTable`). Token strings (`Rgce`/`RgbExtra`, §2.5.98) to the public formula tree and back. 4. **The package** (`XlsbBook`). Keeps the binary package as read, and a SpreadsheetML view of it (`Hub`) that the workbook model works on. Parts with no SpreadsheetML counterpart here (pivot caches and tables, connections, external links, the calculation chain, the worksheet binary index, `customXml`, the VBA project) are not in the view; they stay in the binary package and are written back as they were. ## Nothing is dropped silently A record the translators do not model becomes `base64` (namespace `urn:onedocument:xlsb`, declared `mc:Ignorable`) at its position in the SpreadsheetML view, or beside `sheetData` (`d="b"` before the first row, `d="e"` after the last) when it stands between rows. Writing the part again puts it back where it was. A feature that is translated as a group (data validation, conditional formatting, a filter column, a table) is translated whole or not at all: if one record in the group is not understood, the whole group stays as raw records. Converting to SpreadsheetML removes these elements and reports `loss.xlsb-records-not-converted` with their count. ## What is translated | Part | Both ways | |---|---| | workbook | sheet list and states, 1904 date system, calculation properties (iteration), defined names (global, sheet-scoped, hidden, future-function names), external-sheet table, workbook views; the macro project and other records stay as stored | | worksheet | cells of every kind (RK, real, string, shared string, rich string, boolean, error, blank, formula cells with their cached result), rows (height, hidden, outline, collapsed, style), columns, sheet properties and tab colour, views (panes, selections, zoom, grid lines), merged cells, hyperlinks, print options, margins, page setup, header and footer, drawing and legacy-drawing relationships, sheet protection, AutoFilter, conditional formatting, data validation, row and column page breaks, ignored errors, table parts | | styles | number formats, fonts, fills, borders, cell and style formats, named styles, differential formats, table styles, colours (indexed, theme, RGB, tint) | | shared strings | plain and rich strings, phonetic runs | | comments | authors, comment text with rich runs | | tables | name, range, header and totals rows, columns (totals function and label, calculated-column and totals formulas), table style, the table's AutoFilter | Filters: list filters with date groups, custom filters, top 10 and dynamic filters. Colour and icon filters keep their records raw. Conditional formatting: cell value, expression, text, blanks and errors, time period, above/below average, duplicate and unique values, top/bottom N, colour scales, data bars, icon sets; rules in the newer extension lists (the `x14` forms) stay raw. Not translated, kept as stored in binary saves and listed in the loss report on conversion: pivot caches and pivot tables, connections and query tables, external links, `customXml`, the calculation chain, scenarios, sort states, phonetic properties, the extension records of newer Excel versions. ## Saving `Save`/`SaveAs` to the same format writes only what changed: a part whose SpreadsheetML view was not edited is copied byte for byte (the raw zip entry); an edited worksheet is written again from its view; styles and shared strings are written again when they changed. An untouched workbook saves with every part identical. When a sheet is written again, the worksheet binary index (`xlBinaryIndex`) and the calculation chain are dropped, because both hold offsets or cell order that no longer hold; Excel rebuilds them. A rich-text run whose font is not in the style sheet adds it after the existing fonts, so existing cell formats keep their indexes. Writing from SpreadsheetML (`XLSX → XLSB`) writes every sheet, the style sheet, shared strings, comments and tables from the view. Shared formulas are written as one formula per cell. Chart sheets, dialog sheets and macro sheets raise `OfficeUnsupportedException` (the binary form of a chart sheet is kept only when the binary package itself is saved). ## Formulas The round trip is AST → tokens → AST; tokens → AST → tokens gives the same bytes. Checked over every function of the function table at its fewest and most arguments, and over the 43,084 formulas of the calc grids. - Relative references in names, conditional formats and validations are offsets from the first cell of the range (`PtgRefN`, `PtgAreaN`); cell formulas hold absolute coordinates plus relative flags (§2.5.98.18 and following). - Newer functions are stored as a call of a defined name `_xlfn.NAME` (function number 255); `@` is `_xlfn.SINGLE`, a spill reference `_xlfn.ANCHORARRAY`, and the decoder turns both back into the operators. A call whose callee is computed (`LAMBDA(x,x+1)(3)`) is a user-defined call whose first operand is the expression. - Structured references (`PtgList`) need the table's name and column names; they come from the table parts. - Error values Excel computes but never stores (`#BLOCKED!`, `#CONNECT!`, `#BUSY!`, `#FIELD!`, `#UNKNOWN!`) have no code; a formula holding one is written approximately (`loss.xlsb-formula-approximated`). - The class of each function argument (value, reference, array) follows the function table; the array-class list for the array functions is our reading, to be settled by Excel. ## Streaming `XlsbStreamReader.Open(path | stream)` and `OpenSheet` give an `XlsbSheetCursor` with the members of the XLSX cursor (`ReadRow`, `ReadCell`, address, style, kind, number, text as UTF-8 or into a caller's span, formula flags, dates). It holds one record at a time; shared strings are loaded once as one pool of UTF-16 text, and `GetText` builds each string once. `Spreadsheet.Read`, `Spreadsheet.ReadRows` and `Office.ExtractText` read `.xlsb` through it (see `spreadsheets-streaming.md`). `Workbook.Open` still converts the records to SpreadsheetML in memory and loads that: a direct record-to-model loader would have to rebuild styles, formulas and every sheet feature a second time, so the cursor is the fast path for values. Reading allocates nothing per cell (checked by `XlsbStreamingTests` against the allocation probe: 4,000 rows of five cells, no bytes beyond the fixed opening cost). ## Defects See `defects.md`: `xlsb.record-truncated`, `xlsb.record-short`, `xlsb.row-invalid`, `xlsb.cell-column-invalid`, `xlsb.formula-unreadable`, `xlsb.no-main-part`, `xlsb.part-unreadable`. Only `OfficeException` leaves the public API: a record the translators cannot make sense of is `xlsb.part-unreadable` with the part name. ## Loss vocabulary (conversions and binary saves) | Code | Count is | Meaning | |---|---|---| | `loss.xlsb-part-not-converted` | parts | Binary-only parts left out when converting to SpreadsheetML | | `loss.xlsb-records-not-converted` | records | Raw records (extension and future records) left out | | `loss.xlsb-part-not-written` | parts | SpreadsheetML parts with no binary form written (pivot tables and caches, connections, query tables, external links, calculation chain, …) | | `loss.xlsb-sheet-element`, `loss.xlsb-workbook-element` | elements | Elements with no binary form written yet | | `loss.xlsb-formula-unparsed` | formulas | Formula text that did not parse; the cell keeps its cached value | | `loss.xlsb-formula-approximated` | formulas | A token the format cannot hold exactly | | `loss.xlsb-date-text` | cells | ISO 8601 date cells (`t="d"`), written as text | | `loss.xlsb-data-table` | cells | Data table formulas, written as their cached values | | `loss.xlsb-dxf-gradient` | formats | A gradient fill in a differential format | | `loss.xlsb-filter-dropped`, `loss.xlsb-filter-element` | filters | An AutoFilter without a range; filter elements with no binary form | | `loss.xlsb-conditional-format-dropped`, `loss.xlsb-conditional-format-rule` | blocks, rules | Conditional formatting without a rule or range; rule types with no binary form | | `loss.xlsb-validation-dropped` | validations | A data validation without a valid range | | `loss.xlsb-table-dropped` | tables | A table without a valid range | ## Deviations and open questions for Excel - Shared formulas are expanded to one formula per cell when a sheet is written again (Excel stores one master and `PtgExp` followers). - `fAnd` of a custom filter: the specification's text says 0 means AND and 1 means OR, against the field's name; we follow the text (LibreOffice reads our file the same way). - Icon-set `fReverse`: the specification's description of the bit is the opposite of its name; we write the bit as set for `reverse="1"`. - The array-class parameter list, the error codes `#SPILL!` (0x2C) and `#CALC!` (0x2D), the `fAbove` bit of above-average rules, and the default font of a rich-text run are our reading and are asked of Excel with the generated workbooks. - Generated inputs for Excel are in `testdata/xlsb/` (`manifest.tsv` says what each covers, with its hash); `XlsbProbe generate ` rewrites them. They are not answered yet. # Legacy Excel workbooks (BIFF2 to BIFF8) `RecursiveAI.OneDocument.Spreadsheets.Biff` reads Excel 2.0 to 2003 workbooks and converts them to the SpreadsheetML hub (`Workbook`). Reading follows [MS-XLS] (BIFF8); the layouts of BIFF2 to BIFF5 are not in that specification and are read from the files themselves (see "What is certain" below). `XlsStreamReader` and its cursor read the same files forward-only, without building the workbook, for large files and for value-only jobs; `Spreadsheet.Read`, `Spreadsheet.ReadRows` and `Office.ExtractText` use it. See `spreadsheets-streaming.md`. ```csharp using Workbook wb = Workbook.Open("old.xls"); // sniffs the content, never the extension LossReport loss = wb.ImportLoss!; // what the hub could not carry wb.SaveAs("new.xlsx"); // an ordinary workbook from here on Office.Convert("old.xls", "new.xlsx"); string text = Office.ExtractText("old.xls"); ``` `XlsConverter.ToWorkbook(source, context, loss, token)` is the same with an explicit loss sink. The converted workbook has no file path, so `Save()` is refused and `SaveAs` picks the format. Opening never recalculates: cached results stay as the file stored them. ## Layers 1. **Records** (`BiffRecordSet`, `Biff` namespace): a stream is cut into records (a 2-byte type, a 2-byte length, up to 8,224 data bytes). Every record is kept as an offset and length over the stream's bytes, unknown and future records included; nothing is copied or dropped at this layer. A record that claims more bytes than the stream holds ends the list (`biff.truncated-record`). `CONTINUE` records are joined where a reader needs them: a string split by a `CONTINUE` restarts with a new flag byte that says whether the rest is 8-bit or UTF-16 ([MS-XLS] §2.5.293). 2. **Conversion** (`XlsConverter`): the first `BOF` gives the version; the workbook-global records (code page, fonts, formats, XFs, styles, palette, shared strings, sheet list, `SUPBOOK`/`EXTERNSHEET`, names) are read first, then each sheet substream in `BOUNDSHEET` order (found by its stored offset; when the offset points nowhere the next unused substream is taken, `biff.sheet-missing` if there is none). 3. **Formulas** (`PtgDecoder`, `PtgEncoder`): `Ptg` tokens ([MS-XLS] §2.5.198) become the public formula tree and print as the A1 text a workbook stores. Shared formulas (`SHRFMLA` with `PtgExp` and relative `PtgRefN`) are resolved to one text per cell; array formulas become an anchor with its range, the other cells keep values; data tables become `dataTable` formulas. The encoder writes a tree back as BIFF8 tokens (references in the reference class, no evaluation hints), so `tree → tokens → tree` is exact; the writer uses it. The function numbers come from `XlsbFunctionTable`, the one table behind the BIFF reader, the BIFF8 writer and the binary workbook codec; BIFF8 uses only the numbers below 380. ## What carries over | Source | Hub | |---|---| | `NUMBER`, `RK`, `MULRK`, `INTEGER` | numbers (RK: the 30-bit integer or the high 30 bits of a double, divided by 100 when flagged) | | `LABELSST`, `LABEL`, `RSTRING` | text (rich-text runs dropped, `loss.rich-text`) | | `BOOLERR`, `BLANK`, `MULBLANK` | booleans, errors, styled empty cells | | `FORMULA` + `STRING` | formula text and the cached result (number, text, boolean, error) | | `XF`, `FONT`, `FORMAT`, `STYLE`, `PALETTE` | `styles.xml`: fonts, fills (all 18 patterns), borders (14 line styles, diagonals), alignment, rotation, indent, shrink, protection, number formats, cell styles. Colour indexes stay indexed (`indexed="n"`) and a custom palette becomes `indexedColors`. Formats below id 164 that the file defines are renumbered into the custom range. | | `ROW`, `COLINFO`, `COLWIDTH`, `DEFCOLWIDTH`, `DEFAULTROWHEIGHT` | row height (only when the file marks it manual), hidden rows and columns, outline levels, default formats | | `MERGEDCELLS` | merged ranges | | `WINDOW2`, `PANE`, `SCL` | grid lines, headers, zeros, right-to-left, zoom, frozen and split panes | | `NAME` | defined names (workbook and sheet scope), built-in names (print area and titles, filter database); function and macro names are dropped | | `HLINK` | hyperlinks (URL, file and in-document targets) | | `NOTE`, `OBJ`, `TXO` | comments (the text from the `TXO` of the note's object) with a plain legacy drawing so the notes show | | `DV` | data validation (all types, operators, messages; typed lists as a literal list) | | `CONDFMT`, `CF` | conditional formatting: "cell value" and "formula" rules with their differential formats (`DXFN`: font, fill, borders, number format) | | `CONDFMT12`, `CF12` | the Excel 2007 rule kinds: colour scales, data bars, icon sets (all 17 sets), top/bottom N and percent, text rules (contains, does not contain, begins, ends), blanks, errors, the date periods, above/below average (with equal and standard deviations), unique and duplicate values, with their `DXFN12` formats and priorities. `CFEx` (the extension of a plain `CF` rule) is not read and counts as `loss.conditional-formats-extended`. | | `SETUP`, margins, `HEADER`, `FOOTER`, centring, grid lines, breaks, `WSBOOL` | page setup, print options, header/footer, page breaks, fit-to-page, outline summary direction | | `PROTECT`, `OBJPROTECT`, `SCENPROTECT`, `PASSWORD` | sheet protection with the legacy 16-bit password hash | | `AUTOFILTERINFO` + `_FilterDatabase` | the filter range (criteria are dropped) | | `DATEMODE` | 1904 date system | | `MSODRAWINGGROUP`, `MSODRAWING` | pictures (JPEG, PNG, DIB as BMP, TIFF, EMF and WMF inflated) with their cell anchors | ## Loss report `Workbook.ImportLoss` lists, with counts, what the hub could not take. Codes are `loss.`: `loss.document-properties` (properties of the summary information streams with no counterpart, named in the message; the rest are carried, see `ole.md`), `loss.ole-objects` (embedded objects that could not be matched to their shape or storage, and linked objects: they keep their picture), `loss.charts` (chart sheets and embedded charts; their records stay in the record layer only), `loss.vba` (the macro project is not carried), `loss.macro-sheets`, `loss.macro-names`, `loss.pivot-tables`, `loss.drawing-shapes` (shapes other than pictures), `loss.form-controls`, `loss.pictures` (a picture whose blip could not be decoded), `loss.rich-text`, `loss.formulas` (formulas that could not be converted keep their cached values), `loss.external-links` (references into other workbooks: the cell keeps its cached value), `loss.hyperlinks` (a target of a kind that is not read), `loss.conditional-formats` and `loss.conditional-formats-extended` (a rule that has no hub form, or `CFEx`), `loss.data-validation`, `loss.autofilter-criteria`, `loss.defined-names`, `loss.document-properties` (on import only), `loss.legacy-cell-borders` (BIFF3 and BIFF4 cell borders and fills). ## Defects Every repair is a `Record` defect located by stream path and record offset, except where noted. See `defects.md` for the table. An encrypted workbook (`FILEPASS`) is decrypted with `OpenContext.Password`, or raises `OfficeEncryptedException`; see `encryption.md`. Any exception that is not an `OfficeException` raised while reading is turned into `biff.malformed`. ## Code pages Eight-bit text in BIFF2 to BIFF5 (and in `FILEPASS`-free BIFF8 header strings) is decoded with the workbook's `CODEPAGE` through the .NET `Encoding` registry. The reader registers `CodePagesEncodingProvider` for this. A code page the runtime does not have is reported as `biff.codepage-unavailable` (kind `Encoding`) and the text is read as Windows-1252. BIFF8 compressed strings are Latin-1 by definition and do not depend on the code page. ## What is certain, and what is not - **BIFF8** is read from [MS-XLS] (v20250819) and checked against files LibreOffice writes from generated inputs (`testdata/xls`: values, formulas, styles, names, merges, comments, validation, conditional formats, pictures; the test compares the converted workbook with the workbook the file was written from). - **BIFF5** record layouts (XF, FONT, NAME, FORMULA, BOUNDSHEET, EXTERNSHEET, narrow references) are written from the format as it appears in files; LibreOffice no longer writes BIFF5, so they are covered by hand-built streams and by the govdocs1 corpus read on the build host (counts in the release notes), not by an oracle. The XF alignment, border and fill bit positions of BIFF5 are the least certain part. - **BIFF2 to BIFF4**: values, text, booleans, errors, formulas, fonts, number formats and column widths. BIFF2 cells carry their format in three attribute bytes (folded into the XF table); BIFF3 and BIFF4 XF records are read for font, format, protection and horizontal alignment only. - Excel's own conversion of the same legacy files is the reference for open questions: the cases are listed in `testdata/xls/excel-cases.md` and are answered at the release gate. ## Not converted yet Chart conversion, VBA, pivot tables, the `CFEx` conditional-format extension, external workbook references as formulas, drawing shapes other than pictures, form controls, filter criteria, phonetic text, and the OLE objects (kept as opaque storages in the record layer's source file, not in the hub). ## Writing ```csharp workbook.SaveAs("out.xls"); // the extension picks BIFF8 Office.Convert("in.xlsx", "out.xls"); LossReport loss = workbook.ExportLoss!; // what the last .xls save could not carry ``` `XlsWriter` ([MS-XLS] §2.1.7.20) writes one clean BIFF8 dialect into a compound file with a single `Workbook` stream (class id of an Excel 97-2003 workbook, code page 1200). Everything is rebuilt from the hub; nothing of a legacy file's own record stream is kept, so records the hub has no place for (chart records, OLE storages, phonetic text, `CFEx`) are not carried and the loss report says so where they were counted at read time. **Written:** the workbook globals (`INTERFACEHDR`, `CODEPAGE`, `WINDOW1`, `DATE1904`, fonts, formats, XFs, styles, palette, `BOUNDSHEET8`, `SUPBOOK`, `EXTERNSHEET`, names, `SST` and `EXTSST` with `CONTINUE` splitting at string boundaries), and per sheet: `INDEX` and `DBCELL` blocks of 32 rows, `ROW`, `NUMBER`/`RK`/`LABELSST`/`BOOLERR`/`BLANK`/`FORMULA` (+ `STRING`, `ARRAY`, `TABLE`) records, `COLINFO`, `DEFCOLWIDTH`, `MERGEDCELLS` (1,026 to a record), `WINDOW2`, `PANE`, `SELECTION`, `SCL`, page setup and print options, breaks, header and footer, sheet protection with the 16-bit password hash, the autofilter range with its hidden `_FilterDatabase` name, hyperlinks (URL, in-document), data validation, conditional formats, and cell notes (`MSODRAWING` with the shape, `OBJ`, `TXO` and `NOTE`, with a `MSODRAWINGGROUP` in the globals). Excel 16.0.20527 opens workbooks written with notes (`docs/office-oracle-1.0.md`, defect 4): the declared lengths of a note's shape container and of the containers above it count the eight-byte text box marker that sits in the next `MSODRAWING` record. - **Styles.** Every hub cell format becomes one XF (flattened, with the cell style as parent); the style formats come first, with at least 15 as Excel expects. Colours are palette indexes: a colour that is already in the default palette keeps its index, others take free slots of a custom `PALETTE` (56 slots), beyond that the nearest is used (`loss.colors`). Gradient fills become a solid fill of the first stop (`loss.gradient-fills`). Fonts, formats and XFs beyond BIFF8's counts take the first font, General and the default format (`loss.fonts`, `loss.number-formats`, `loss.cell-formats`). - **Formulas** go through the formula tree and `PtgEncoder`; shared formulas are written out as one formula per cell. Array formulas keep their range (a dynamic array becomes a legacy array over its current range, `loss.dynamic-arrays`); data tables are `TABLE` records. A formula BIFF8 cannot express (structured references, `LET`, `LAMBDA`, newer functions without a `_xlfn.` form, references into other workbooks, over 8,192 token bytes) keeps its cached value as a constant (`loss.formulas`). A name that cannot be encoded stays in its place and refers to `#REF!` (`loss.defined-names`), because formulas number names by position. - **Conditional formats.** A group of up to three "cell value"/"formula" rules is written as `CONDFMT` + `CF`; any other group as `CONDFMT12` + `CF12` ([MS-XLS] §2.4.43, §2.4.57) with colour scales, data bars, icon sets, top/bottom, text, blank, error and date rules. Unique, duplicate and average rules, and the Excel 2010 extension kinds, are `loss.conditional-formats-extended`. - **Limits.** 65,536 rows and 256 columns: cells, column settings, merges, comments and references outside are not written (`loss.cells-outside-grid`, `loss.formulas`). Strings are cut at 32,767 characters, sheet names at 31. - **Not written, reported:** charts and chart sheets (`loss.charts`), pictures (`loss.pictures`; the shared OfficeArt layer has no writer yet), drawing shapes, tables (`loss.tables`), sparklines, pivot tables, external links, macro projects, rich-text runs (`loss.rich-text`), first/even headers and footers, tab colours, sheet protection permissions beyond the three flags, modern password hashes, hyperlink screen tips, autofilter criteria, extended data validation. Document properties are written: the core, extended and custom properties become the `\u0005SummaryInformation` and `\u0005DocumentSummaryInformation` streams (`ole.md`). - **Password.** `Workbook.SaveAs(..., OfficeFormat.Xls, password)` encrypts the stream with RC4 CryptoAPI (`encryption.md`). - **Left for the Excel check:** whether Excel opens the output without repair (the cases are in `testdata/xls/excel-cases.md`). # OpenDocument spreadsheets and OpenFormula ODS, OTS and flat FODS (OpenDocument 1.2 and 1.3) read into the workbook hub and write back out, with a loss report each way. The package layer (`OdfPackage`, `odf.md`) does the zip, manifest and flat-file work; this page covers the spreadsheet conversion in `RecursiveAI.OneDocument.Spreadsheets.Odf` and the formula language in `RecursiveAI.OneDocument.Calc.OpenFormula`. Defect codes (`ods.*`, `openformula.*`) are in `defects.md`. ## Using it `Workbook.Open` sniffs the content (a `mimetype` of `application/vnd.oasis.opendocument.spreadsheet` or `-template`, or an `office:document` root for flat files), so the extension never decides. `SaveAs` picks `.ods`, `.ots` or `.fods` by extension, or takes the format explicitly; `Office.Convert` goes both ways between these and every other workbook format. `OpenDocumentSheet.ToPackage` is the public converter from ODF bytes to an in-memory XLSX package, for callers who want the OOXML parts. An ODS that is opened and saved without an edit writes the bytes it was read from. Any edit ends that, and the file is written by the exporter. Opening never recalculates: the values the file stores stay until `Recalculate`. ## Reading (ODF → hub) The importer converts the document to an in-memory XLSX package that `Workbook` wraps, as the other legacy readers do. | ODF | Hub | |---|---| | `table:table` | sheet (hidden, tab colour, right-to-left, grid lines, frozen panes and active sheet from `settings.xml`) | | `table:table-column`, `table:table-row` | widths, heights, hidden and filtered rows and columns, outline groups, header rows and columns; repeats within the limits | | `table:table-cell`, `covered-table-cell` | cells of the types float, percentage, currency, date, time, boolean and string with their cached values; spans to merged ranges; array formulas | | `style:style` (cell, column, row, table) | cell formats: font, fill, border, alignment, wrapping, rotation, protection; a style's parent chain is flattened | | `number:*-style` | Excel number-format codes through the shared number-format service, including conditions, colours, fractions, scientific, text parts | | `table:named-range`, `named-expression` | defined names (workbook and sheet scope); `Excel_BuiltIn_*` names to built-in names (print areas and titles, filter databases) | | `table:database-range` | auto-filters (without conditions) | | `calcext:conditional-format`, `style:map` | conditional formats: cell-value and formula conditions with differential formats, colour scales, data bars, icon sets | | `table:content-validations` | data validation (list, whole, decimal, date, time, text length, custom) with messages | | `office:annotation` | comments | | `text:a` in a cell | hyperlinks | | `style:page-layout`, `master-page`, print ranges | page setup, margins, scaling, headers and footers, print areas and titles | | `office:meta` | document properties | Reported as losses (`loss.*` on `Workbook.ImportLoss`): shapes, pictures and charts on a sheet and body content without a counterpart are reported under the element's name (`loss.shapes`, `loss.data-pilot-tables`, `loss.scenarios` and so on), then `sparklines`, `tracked-changes`, `protection-password` (the hash is a different algorithm), `filter-conditions`, `database-ranges` (a named range keeps only its cells), `rich-text` (formatting inside a cell's text; the text is kept), `number-format` (parts of a data style Excel codes cannot express), `conditional-formats` (rules with no Excel counterpart), `validation`, `null-date` (a date origin other than 1899-12-30 is converted) and `formulas` (see below). Formulas are read as OpenFormula. A formula that does not parse, uses a syntax other than `of:` (`oooc:`, `msoxl:`), or points at another document keeps its cached value as a constant and reports `ods.bad-formula` or `ods.formula-dialect` with the cell address. Limits: repeats are counted against the cell budget from `OfficeLimits` (`odf.repeat-capped`); rows and columns past the sheet's size are dropped (`ods.row-limit`, `ods.column-limit`); an empty repeated row or cell costs nothing. ## Writing (hub → ODF) The exporter reads the hub through its public model and each sheet's markup frame, and writes `content.xml`, `styles.xml`, `meta.xml` and `settings.xml` (view settings: active sheet, freeze panes, zoom) through `OdfPackage.Create`. Flat output writes the same four parts into one `office:document`. Every format written is checked against the ODF 1.3 RelaxNG schema (see Checks). Styles are written as automatic styles, de-duplicated; number formats become `number:*-style` elements with the same meaning; formulas are printed as OpenFormula with `of:=`; the cached value is written beside every formula (`office:value`, `office:string-value`, `calcext:value-type="error"` on error cells). Reported as losses (`ExportLoss`): `charts`, `pictures`, `drawings`, `pivot-tables`, `vba`, `external-links`, `tables` (the table definition; its cells and styles are kept), `validation` (a rule ODF cannot express), `formulas` (dynamic arrays and spill, `LAMBDA`, structured references, data tables: the cached value is written), and `function-not-portable` (a function LibreOffice does not know is written anyway, and other readers show `#NAME?` after recalculating). The content part is built in memory before it is written, so a very large sheet needs memory in proportion; `MaxPartSize` bounds it. ## OpenFormula `OpenFormulaParser.TryParse(text, out node, out error)` reads OpenDocument 1.3 Part 4 into the public formula AST; `OpenFormulaPrinter.Print(node)` prints an AST back. Both ends use the same AST as the A1, R1C1 and BIFF forms, so a formula moves between notations without a separate model. Syntax handled: - the `of:=` prefix is stripped; `;` separates arguments, inline arrays use `;` between columns and `|` between rows (`{1;2|3;4}`), `~` is union, `!` is intersection; - references are bracketed: `[.A1]`, `[.$A$1:.B2]`, `[$Sheet1.A1:.B2]`, `[$'My Sheet'.A1]`, whole columns and rows (`[.A:.A]`, `[.1:.3]`), 3D ranges (`[$S1.A1:$S3.A1]`); a reference into another document is `openformula.external-reference`; - errors: `#N/A`, `#REF!` (printed lower-case `#ref!` as LibreOffice does), `#DIV/0!`, `#NAME?`, `#NULL!`, `#NUM!`, `#VALUE!`; - `TRUE()`/`FALSE()`, percent, unary minus precedence as Excel (`-A1^2` is `(-A1)^2`), string escapes by doubling; - function names with `COM.MICROSOFT.`, `ORG.OPENOFFICE.`, `ORG.LIBREOFFICE.` and `LEGACY.` prefixes in any case; unknown `ORG.*` names are kept as the engine's names. Function names (`OpenFormulaFunctions`): almost all names are the same as Excel's. The differences, taken from the files the reference reader writes (`testdata/ods/openformula-functions.tsv` lists the printed form of every function the engine has, 509 rows; `openformula-expressions.tsv` lists 72 expression forms): | Rule | Examples | |---|---| | `COM.MICROSOFT.` prefix on functions Excel added in 2010 and later | `STDEV.S`, `NORM.DIST`, `CONCAT`, `IFS`, `XLOOKUP`, `FILTER`, `SORT`, `LET`, `TEXTJOIN` | | `LEGACY.` prefix on the pre-2010 statistical names that have replacements | `CHIDIST`, `FDIST`, `NORMSDIST`, `TDIST` | | spelled differently | `F.DIST` ↔ `FDIST`, `F.INV` ↔ `FINV`, `SKEW.P` ↔ `SKEWP`, `FORMULATEXT` ↔ `FORMULA` | Functions the engine has and OpenFormula (as the reference reader knows it) does not: `LAMBDA`, `MAP`, `REDUCE`, `SCAN`, `BYROW`, `BYCOL`, `MAKEARRAY`, `GROUPBY`, `PIVOTBY`, `PERCENTOF`, `REGEXEXTRACT`, `REGEXREPLACE`, `REGEXTEST`, `TRIMRANGE`, `ARRAYTOTEXT`, `VALUETOTEXT`, `BAHTTEXT`, `PHONETIC`, `TRANSLATE`, `DETECTLANGUAGE`, and a few complex-number and `BINOM.DIST.RANGE` functions. The printer writes them unchanged and reports `openformula.function`, which the exporter turns into `loss.function-not-portable`. Functions with the same name and a different result or argument order are not known to exist in the shared set; the 509-row grid is the evidence. The printer cannot express, and reports as not portable: spill references (`A1#`), structured references, member paths, `LAMBDA` invocation, `@`. The exporter then writes the cached value without a formula and reports `loss.formulas`. ## Deviations - Cell styles are flattened: a cell's parent-style chain becomes one cell format, so a round trip through the hub does not keep the style hierarchy. - A space-preserving `_)` in an accounting-style number format has no counterpart in ODF and becomes a space. - `calcext:value-type` is written only on error cells (the one extension attribute in an otherwise strict file); `tableooo:tab-color` carries the tab colour. Both are accepted by the reference reader and ignored by strict ODF 1.3 readers. - Formulas in a syntax other than `of:` are not read. - Charts, pictures and shapes are loss items; the cell values they were built on are kept. ## Checks - `tools/ods/make_inputs.py` and `make-oracle-fixtures.sh` produce the oracle fixtures (`testdata/ods`, LibreOffice 26.8.2.2 as a black box; see `oracle-version.txt`). - `tools/ods/run-oracle-check.sh` converts the workbooks in `testdata` to ODS and FODS, validates them with Jing against the ODF 1.3 schema (`tools/validate/validate_odf.py --ignore-extensions`: the `calcext:` and `loext:` extension attributes are not part of the schema), has the reference reader convert each file back to XLSX and compares values and formulas with the source (`OdsOracleTests`). - Fuzz targets `ods` (the reader and the exporter) and `openformula` (the parser and printer). # Presentations: the PresentationML hub `RecursiveAI.OneDocument.Presentations` holds `Presentation`, `Slide`, `SlideLayout`, `SlideMaster`, `NotesSlide`, `NotesMaster`, `Shape`, `TextBody`/`Paragraph`/`Run` and `Table`. It is a set of views over the lazy markup tree of each part; nothing is copied out of the part buffers. The XML documentation on the types states the details; this note states the rules. ## What is read, when `Presentation.Open` (path, stream or `IByteSource`) identifies the content (never the extension), opens the package and reads the content types, the package relationships and the presentation part. Every other part is read on first use. | Part | Read when | |---|---| | the presentation part | the first member that needs it (lazy main part); an untouched save never parses it | | a slide, layout, master, notes slide | first use of that object's shapes or properties, or when a slide operation needs its relationships | | theme | the first colour or font resolution through that master or layout | | comments, comment authors, sections | when they are first enumerated | | media, charts, diagrams, OLE, tags, timing, transitions | never; they stay as stored | Supported package kinds: PPTX, PPTM, POTX, PPTX templates and slide shows (the type is retyped through the main part's content type on save), Transitional and Strict, and Flat OPC. A chart, SmartArt diagram or OLE object is a graphic frame whose `References` lists the related parts; the parts stay as stored. `mc:AlternateContent` is kept as written; the shape children of an alternate content block are those of its first `Choice`, and edits never touch the `Fallback`. ## The round-trip contract - A part nobody edited is copied as stored (the raw zip entry), so a presentation opened and saved without edits has every part byte-identical. - An edited part keeps every element, attribute, prefix, `mc:` block and `extLst` it did not edit, in order. - `docProps/app.xml` (slide counts, titles) is not rewritten by slide operations; PowerPoint rebuilds it on save. ## Inheritance Resolution follows the chain slide → layout → master → presentation defaults. - **Placeholder matching.** Slide to layout: by `idx` when it is non-zero (a type-compatible placeholder preferred), otherwise by type, where `title` and `ctrTitle` match each other. Layout to master: by role; every content type and `subTitle` map to the master's body placeholder. Notes slide to notes master: by type and `idx`. - **Text styles.** For a paragraph at level *n*, properties come from, in order: the shape's own `a:lstStyle`, the matching layout placeholder's `a:lstStyle`, the matching master placeholder's `a:lstStyle`, then the master's `titleStyle`, `bodyStyle` or `otherStyle` by role (a non-placeholder shape skips the master styles and uses `otherStyle`), then the presentation's `defaultTextStyle`. Paragraph and run properties merge attribute by attribute; the bullet (none, character, auto-number, picture, with its font, colour and size) is taken as a unit from the first source that names one. - **Body properties and bounds** inherit the same way (insets, anchor, wrap, autofit kind, and the shape's offset and extent when the slide's `xfrm` is absent). - **Colour.** `p:clrMap` of the master, overridden by the layout's and slide's `p:clrMapOvr`, maps `bg1`/`tx1`/… to theme slots; scheme colours resolve through the map and then the theme of the master. Placeholder matching and the chain are written from ECMA-376 Part 1 §19.3 and §21.1; PowerPoint's own behaviour for odd decks is not yet checked against the application. ## Editing All operations are on `Presentation`; each leaves untouched parts alone. - **AddSlide(layout)** creates a slide part with the layout's non-footer placeholders (date, footer, slide number and header placeholders are skipped; a text body only for title, centre title, subtitle, body and object placeholders), the relationship to the layout, the `sldIdLst` entry with a new id, and the content-type override. - **DuplicateSlide** copies the slide part and the parts only it uses. Shared (the new slide's relationship points at the same part): images, hyperlinks, layouts, masters, notes and handout masters, themes, video, audio, media, HD photo, fonts. Copied transitively with new names: charts and their embedded workbooks, diagrams, OLE objects, tags, comments and the notes slide. `p14:creationId` is replaced by a deterministic hash of the source's value and the new part name. The copy joins the source slide's section. Duplicated markup is the source markup, so schema errors the source slide had are present in the copy and nothing else is added. - **DeleteSlide** removes the slide, its `sldIdLst` entry and its section entry, and every part only it reached (decided by a fixed-point reference count over the package); shared parts stay. - **MoveSlide** reorders `sldIdLst` and the section lists. - **SetNotesText** edits the notes slide, creating it (with its relationships to the slide and the notes master) when the slide has none. - **Text edits.** Setting a shape's text edits its first paragraph in place; further paragraphs copy the first paragraph's `a:pPr` and first run's `a:rPr`. Characters XML cannot hold are written as `_xHHHH_`; a vertical tab is a line break (`a:br`). - **Table cells** are addressed on the logical grid; `gridSpan`, `rowSpan`, `hMerge` and `vMerge` give the merge model. Slide ids come from the existing maximum plus one; names and relationship ids are the next free ones. Output is deterministic. ## Defects Codes `pptx.*` in `defects.md`. A missing or wrong-root slide, layout or master reads as empty or as inheriting nothing and is reported once per part; a missing main part is fatal. ## Rendering Slides, notes pages and handouts render to PDF and PNG through `PresentationRenderer` in the rendering package; the rules are in `presentation-rendering.md`. ## Legacy `.ppt` PowerPoint 97-2003 files open as presentations through a conversion to PresentationML with a loss report; see `ppt.md`. # PowerPoint 97-2003 files (`.ppt`) `Presentation.Open`, `Office.Open`, `Office.ExtractText` and `Office.Convert` read PowerPoint binary presentations. The file is read at record level (`RecursiveAI.OneDocument.Ppt.Binary`, on the shared OfficeArt reader in `officeart.md`), converted to PresentationML in memory, and the presentation hub opens that package as typed views. Everything the hub does with a `.pptx` it does with the converted file: edit, render, template, save as `.pptx`. `Presentation.ImportLoss` lists what did not map; `Presentation.SourceFormat` is `OfficeFormat.Ppt`. Writing `.ppt` is not offered. The result has no file path, so `Save()` never writes over the binary original; `SaveAs` writes PresentationML. ## Structure that is read | Stream | Use | |---|---| | `PowerPoint Document` | the records: persist objects, persist directories, user edits | | `Current User` | the offset of the newest user edit; header token `0xF3D1C4DF` marks an encrypted file | | `Pictures` | the BLIP data the drawing group's store points into | | anything else | listed in `PptFile.StreamNames`; a `_VBA_PROJECT_CUR` storage is reported (`ppt.vba-not-carried`), never run | **Edit chain and persist directory** ([MS-PPT] §2.1.2, §2.3.3 to §2.3.6). PowerPoint appends to the stream on every incremental save: the objects that changed, a `PersistDirectoryAtom`, and a `UserEditAtom` that names its predecessor. The reader starts at the edit the Current User stream names and follows `offsetLastEdit` back while the offset strictly decreases. The persist directory of each edit is read newest first and the first definition of a persist identifier wins, so the last edit wins and older copies stay in the stream untouched. A persist object is parsed when first asked for and cached. Repairs, each a defect with the stream offset: - no usable Current User stream, or an offset that is not a `UserEditAtom`: the edits are found by scanning the top-level records; - a chain that does not move backwards ends at the loop; - a user edit whose directory is not there: its objects are ignored, older edits still serve; - a document persist identifier that is not listed: the first top-level `DocumentContainer` is used; - a record that claims more bytes than the stream holds is clamped (`art.record-overrun`). Encrypted files (header token or a `CryptSession10Container` reference) are decrypted with `OpenContext.Password`, or raise `OfficeEncryptedException`; see `encryption.md`. PowerPoint 4.0 and 95 files (a `PersistentStorage Directory` stream, no user edits) are `OfficeUnsupportedException`: [MS-PPT] does not describe them. ## Text A text body is a `TextHeaderAtom` and the records after it: `TextCharsAtom` (UTF-16) or `TextBytesAtom` (Latin-1), `StyleTextPropAtom`, metacharacter atoms (slide number, date, header, footer), `TextSpecialInfoAtom` (language), `TextRulerAtom`, interactive records. Paragraphs end at U+000D, forced line breaks are U+000B; every body has an implicit terminating break, so style runs cover the text plus one. Bodies live in the shape's `OfficeArtClientTextbox`, or in the document's slide list when the shape holds an `OutlineTextRefAtom` (the older layout). `TextPFException` and `TextCFException` are read with their mask bits; a field exists only when its bit is set, and what is not set inherits from the master text styles, exactly as in PowerPoint. The converter writes only the set fields, so the hub's inheritance resolves them the same way. `TextRulerAtom` supplies the left margin and indent per level that paragraph runs cannot carry. Paragraph spacing: 0 to 13200 is a percentage of the line, a negative value is master units. Master units are 1/576 inch (1,587.5 EMU). ## Conversion - **Masters and layouts.** Each main master becomes a slide master with a theme, `txStyles` from its `TextMasterStyleAtom` records (title, body, other) and its drawing. PowerPoint 97-2003 has no layouts; one layout is written per distinct slide layout type and placeholder set used with a master, modelled on the first slide that uses it (placeholders only, positions from that slide). A title master becomes the "Title Slide" layout of its main master, with its shapes. Layout type names follow `SlideLayoutType`. - **Colours.** The eight scheme colours become the theme (`dk1` text, `lt1` background, `dk2` shadows, `lt2` title text, accents 1 to 4 fill, accent, accent and hyperlink, accent and followed hyperlink). A scheme index becomes a theme colour when the slide's scheme equals its master's, otherwise the slide's RGB value. - **Slides.** Shapes keep their z-order, groups become groups with the group's coordinate space as child extents. Rotation and flips come from the OfficeArt properties. Backgrounds (the drawing's background shape) are written unless the slide uses the master's. Slide flags: follow master objects is `showMasterSp`, hidden comes from the transition record. - **Geometry.** Preset shape types (about 130) map to DrawingML presets; freeforms become custom geometry from the vertex and segment arrays; the rounded rectangle's corner is converted; other adjustments are dropped (reported). - **Fill, line, shadow.** Solid, picture, texture and gradient fills; dash styles, compound lines, caps, joins, arrowheads; an outer shadow. Pictures come from the BLIP store (`Pictures` stream), metafiles inflated, DIBs given a BMP header; crops become `srcRect`. - **Text boxes and placeholders.** Insets, anchors, wrap, text flow and fit-shape-to-text come from the text properties. Placeholder types map to `ph` types with indexes assigned in order (date 10, footer 11, slide number 12). WordArt text (geometry text) is written as plain text. - **Fields.** Slide numbers and dates become `a:fld`; a header or footer metacharacter becomes the text of the page's `HeadersFooters` record, else the document's. - **Tables** as PowerPoint 2003 stores them: a group whose properties mark it a table and whose children are cell rectangles in row-major order; the row count is the length of `tableRowProperties`. A regular grid becomes `a:tbl` with each cell's own outline as its borders; anything else stays a group. - **Notes** become notes slides with a notes master, the handout master a handout master, each with a theme. - **Comments** (PowerPoint 2002 extension data) become legacy comment parts with an author list; positions are master units, as in `p:pos`. - **Hyperlinks** on shapes and text ranges: web and file targets become external relationships, `slideId,slideNumber,title` targets slide jumps, jump actions (next, previous, first, last, end show) the matching `ppaction`. - **Transitions** map by effect type and direction; advance on click, automatic advance time and speed are carried. Effects PowerPoint 97 had that PresentationML lacks are reported. ## Loss report `Presentation.ImportLoss` items (code, feature, count, message); each is also a defect with the record offset in the source stream. | Code | What did not carry | |---|---| | `ppt.shape-approximated` | a shape type with no preset (or a picture frame with nothing in it) is a rectangle | | `ppt.shape-adjust-dropped` | adjustment handles other than the rounded rectangle's corner | | `ppt.fill-approximated` | pattern fills are solid fills in the foreground colour | | `ppt.path-escape` | arcs and fill switches in freeform paths | | `ppt.geometry-guides` | guide-formula coordinates in freeform paths (written at the origin) | | `ppt.wordart-flattened` | WordArt effects and text paths (text kept) | | `ppt.picture-pict` | a Macintosh PICT picture is carried as is; Office will not show it | | `ppt.picture-missing` | a picture index that is not in the store | | `ppt.table-as-group` | a table group that is not a regular grid | | `ppt.table-borders-approximated` | table border lines drawn as separate shapes | | `ppt.transition-dropped` | a transition type PresentationML lacks | | `ppt.animations-dropped` | animation effects (shape animation info, PowerPoint 2002 time nodes) | | `ppt.ole-object-flattened` | OLE objects whose storage or picture is missing or unreadable, and controls: only the preview picture is written. Objects that could be carried are not counted: see `ole.md` | | `ppt.document-properties-not-carried` | properties of the summary information streams with no counterpart in the package (the thumbnail, the VBA signature, hyperlink lists, custom types without an equivalent), each named in the message, or a stream that could not be read. The rest are carried: see `ole.md` | | `ppt.vba-not-carried` | the VBA project | | `ppt.header-footer-field` | a header or footer metacharacter with no text to show | | `ppt.placeholder-unknown` | a placeholder type this converter does not know | ## Open questions The specification leaves these open; PowerPoint decides at the final gate. `testdata/ppt` holds LibreOffice's PowerPoint 97 output for the generated decks and `tools/office-oracle` cases ask PowerPoint to open each file and save it as `.pptx` for comparison; the answers are not in yet. - gradient angle: `fillAngle` is counter-clockwise from the upward vector; the conversion measures DrawingML's clockwise angle from the x axis as 270 degrees minus `fillAngle`; - the ribbon pair (shape types 53 and 54, 107 and 108) maps to `ribbon2`/`ribbon` and `ellipseRibbon2`/`ellipseRibbon` by position, not by a verified correspondence; - first-line `indent` is written relative to the left margin (`indent - marL`) as DrawingML defines it; - layout modelling: one layout per layout type and placeholder set, not the layouts PowerPoint itself would generate; - table borders from the cell outlines, not from the separate line shapes. ## Limits The limits object applies throughout: `MaxRecordCount`, `MaxRecordSize`, `MaxRecursionDepth` (record nesting and group nesting), `MaxAllocationBytes` (inflated metafiles), `MaxPartSize` (the streams), `MaxDefects`. Record lengths are checked against the data before anything is allocated. ## Tools `tools/PptProbe` opens, dumps, converts and lists streams; `tools/ppt/make-fixtures.sh` makes the LibreOffice fixtures, `tools/ppt/check.sh` converts a directory and runs the schema and LibreOffice checks; `fuzz` target `ppt`. # OpenDocument presentations (ODP, OTP, FODP) `Presentation.Open` accepts ODP, OTP and flat FODP by content (the `mimetype` entry or `office:mimetype`), never the extension. The file is converted to an in-memory PPTX package, and the hub wraps it like any PPTX; `Presentation.Format` stays `Odp`, `Otp` or `Fodp`, and `Presentation.ImportLoss` lists what did not map. `SaveAs("x.odp" | "x.otp" | "x.fodp")` (or with an explicit `OfficeFormat`) writes OpenDocument from the hub, and `Presentation.ExportLoss` lists what it could not carry. `Office.Open`, `Office.ExtractText`, `Office.Convert` and the renderer reach the formats through the registry like any other presentation. Direct entry point: `OpenDocumentPresentation.ToPackage` for the PPTX package. The package layer (`OdfPackage`, styles, metadata) is the one in `odf.md`. **Untouched files.** An ODP/OTP/FODP read and saved in its own format without an edit writes the bytes it was read from, so everything the hub does not carry (animations, transitions, charts, embedded objects, producer extensions) survives. Any edit reaches the exporter, which writes a valid package of its own from the hub; a save to another OpenDocument format goes through the exporter too. ## Reading (ODP → hub) | ODF | Becomes | |---|---| | `style:master-page` with its page layout | one `p:sldMaster` per master page (own theme), with its background (`draw:style-name` drawing-page style), drawing content and text styles (`-title`, `-outline1…5` presentation styles) | | `presentation:presentation-page-layout` used by a slide | one `p:sldLayout` per master page and page layout a slide uses: type from the objects it lists (title + subtitle → `title`, title + outline → `obj`, title only → `titleOnly`, nothing → `blank`), placeholders with the geometry the page layout gives | | `draw:page` | `p:sld`: name, master/layout, `presentation:visibility="hidden"` → `show="0"`, background from the page's drawing-page style | | frames with `presentation:class` `title`, `subtitle`, `outline`, `text` | placeholders `title`, `subTitle idx=1`, `body idx=1,2…` (in order of appearance, matching the layout) | | `date-time`, `footer`, `page-number` frames of the master page | placeholders on the slide when the page style turns them on (`presentation:display-footer` and friends); the footer and date text come from the `presentation:footer-decl` / `date-time-decl` the slide names | | `presentation:notes` | a notes slide (body placeholder with the `notes` frame's text) and one notes master | | `draw:rect`, `ellipse`, `circle`, `caption`, `custom-shape` | `p:sp` with a preset geometry: `ooxml-` types (the adjust values are the `draw:modifiers`), `mso-spt`, `flowchart-*` and the standard names (`rectangle`, `round-rectangle`, `right-arrow`…) map; `draw:rect` with a corner radius is `roundRect` | | `draw:polygon`, `polyline`, `path` | `a:custGeom` from `draw:points` / `svg:d` and `svg:viewBox` (M L H V C S Q T Z, absolute and relative; arcs become straight lines) | | `draw:line`, `draw:connector` | `p:cxnSp` (`line`, `straightConnector1`, `bentConnector3`, `curvedConnector3`) | | `draw:g` | `p:grpSp` with the box of its children | | `draw:frame` with `draw:text-box` | a text box or placeholder; with `draw:image` a `p:pic` (PNG, JPEG, GIF, BMP, TIFF, EMF, WMF; zip entries or inline `office:binary-data`); with `table:table` a graphic frame | | `draw:transform` (`rotate`, `translate`) | `rot` and the position of the unrotated box (ODF rotates about the top-left corner, PresentationML about the centre) | | graphic and presentation styles | solid, gradient (linear, axial, radial as a two or three stop gradient) and bitmap fills with opacity; lines (width, colour, dash, caps, joins, arrows); shadows (offset, blur, colour, opacity); text insets, anchor, no-wrap, shrink-to-fit (`a:normAutofit`) and grow-to-fit (`a:spAutoFit`) | | `text:p`, `text:h`, `text:span`, `text:a`, `text:s`, `text:tab`, `text:line-break`, `text:page-number`, `text:date` | paragraphs, runs, hyperlinks (external only), line breaks, slide-number and date fields; white space collapses as Part 1 §6.1.2 says | | text and paragraph properties | size, bold, italic, underline, strike, colour (`use-window-font-color` takes black or white to contrast with the fill), highlight, fonts (latin, Asian, complex through the font-face declarations), super/subscript, caps, spacing, language; alignment, margins and indent, line height, spacing before and after, right-to-left | | `text:list` with list styles | bullets (`a:buChar`, with font, size and colour) and numbering (`a:buAutoNum` from `style:num-format`, prefix, suffix and `text:start-value`); level geometry from the label-alignment or the older space-before / minimum label width form; nesting is the paragraph level. Paragraphs outside a list get `a:buNone`, so they do not pick a bullet up from the master | | `table:table` | `a:tbl` with column widths and row heights from the styles, spans (`gridSpan`, `rowSpan`, `hMerge`, `vMerge` for covered cells), header rows (`firstRow`), cell fill, borders, padding and vertical alignment; repeat counts are capped | | `meta.xml` | `docProps/core.xml`: title, subject, creator, keywords, description, last modified by, dates, language | Every shape is written with direct formatting resolved from its style chain (the shape's own style, the presentation or graphic style it names, the frame's text style, the paragraph and span styles, then the family defaults), so the PPTX does not rely on inheritance that a master page cannot express. ### Reported as losses (`ImportLoss`) | Code | What | |---|---| | `loss.odp-animations`, `loss.odp-transitions` | animations (`anim:*`) and transitions (`smil:*`, `presentation:transition-*`), counted in slides; they stay in the file for an unchanged save | | `loss.odp-embedded-object`, `loss.odp-object`, `loss.odp-object-ole`, `loss.odp-plugin`… | charts, formulas and other embedded objects (the preview picture is kept when there is one), plug-ins, controls, measure lines | | `loss.odp-custom-geometry` | custom shapes whose geometry is a formula-driven path: drawn as rectangles | | `loss.odp-fill-hatch` | hatch fills | | `loss.odp-picture-crop`, `loss.odp-picture-format`, `loss.linked-images`, `loss.pictures` | cropping (it needs the picture's size, and the picture is never decoded), StarView and other formats PresentationML cannot hold, linked pictures (never fetched), pictures the file does not hold | | `loss.odp-internal-links`, `loss.odp-notes`, `loss.odp-comments`, `loss.odp-ruby`, `loss.odp-inline-shapes`, `loss.odp-picture-bullets`, `loss.odp-caption`, `loss.odp-ellipse-*`, `loss.odp-path-arcs`, `loss.odp-regular-polygon`, `loss.odp-line-text`, `loss.odp-page-sizes` | links to slides, footnotes, comments, ruby, shapes anchored in text, picture bullets, caption callout lines, partial ellipses, path arcs, regular polygons, text on lines, master pages whose page sizes differ (the first master's size is used) | | `loss.unmapped-markup.` | any other drawing element, counted | ## Writing (hub → ODP) Masters and layouts become master pages (one for each layout, named after it, holding the layout's placeholders with resolved geometry, the non-placeholder shapes of the master and the layout, and the background). Slides become `draw:page` with their master page, name, hidden state, background and notes. Shapes keep their resolved geometry (placeholders take it from the layout), rotation (`draw:transform`), fill, line, shadow and text formatting; presets with no ODF counterpart of their own are `ooxml-` custom shapes with the adjust values as modifiers, and `custGeom` paths are `draw:path`. Bullets and numbers become list styles (one per distinct bullet and level geometry; a numbered list that comes back after other lists continues counting unless the paragraph restarts it). Tables keep column widths, row heights, spans, cell fill, borders and padding; cells with no fill of their own are written transparent, so no reader paints its default table design on them. Fonts are declared in the font-face declarations. Pictures go to `Pictures/` (or inline base64 in FODP). `meta.xml` carries the core and custom properties. ### Reported as losses (`ExportLoss`) `loss.odp-export-transitions`, `-animations`, `-comments`, `-charts`, `-diagrams`, `-ole`, `-graphic-frames`, `-table-styles` (the banding and header colours a table style gives; formatting on the cells is kept), `-header-rows` (written as ordinary rows: LibreOffice drops the rows of `table:table-header-rows` when it opens a presentation), `-crop`, `-flips`, `-group-rotation`, `-path-arcs`, `-pattern-fills`, `-gradient-stops` (first and last colour), `-picture-bullets`, `-numbering-schemes`, `-slide-links`, `-hidden-shapes`, `-unpositioned`, `-unknown-shapes`, `-empty-geometry`, `-deep-groups`. ## Defects `odf.not-presentation` (fatal), `odf.damaged` (fatal), `odf.export-failed` (fatal), `odp.no-master-page`, `odp.missing-master-page`, `odp.bad-viewbox`, `odp.bad-path`, `odp.empty-table`, `odp.missing-fill-image`, `odp.group-too-deep` and the shared `odf.repeat-capped`, `odf.missing-picture`, `odf.bad-base64`, `odf.bad-date` (see `defects.md`). The limits object passes through: part count, size, XML depth, recursion depth (nested lists and groups). A table is read up to 50,000 cells. ## Checked against ODF 1.3 RelaxNG with Jing (`tools/validate/validate_odf.py`) on every deck of `testdata/` written as ODP and FODP; LibreOffice opens the written files and exports the same number of pages as it does for the PresentationML deck (`tools/odp/lo-open-check.sh`); the ODP files LibreOffice writes from the same decks (`tools/odp/make-fixtures.sh`, committed in `testdata/odp`) read back with the same slides, text, notes, hidden slides, tables and pictures. PowerPoint opening our ODP, and PowerPoint's own ODP conversion, are unanswered (`testdata/odp/ORACLE-REQUEST.md`). # Templating: the engine core Namespace `RecursiveAI.OneDocument.Templating` in the core package. This page states the template syntax, the extensions to Excel formula syntax, how the engine reads data, the contract a document binder implements, and the defect codes (the full catalogue is in `defects.md`). The engine knows no document format: the DOCX, XLSX and PPTX binders supply text and rebuild documents from what the engine returns. The syntax is Liquid/Jinja-style blocks around Excel-formula expressions. It is written in one place (`TemplateSyntax`) and separate from the renderer, so the surface can change without touching evaluation. ## Syntax | Form | Meaning | |---|---| | `{{ expression }}` | prints the value of an expression | | `{% for item in expression %} … {% else %} … {% endfor %}` | repeats the body for each item; `else` runs when there are none | | `{% if expr %} … {% elif expr %} … {% else %} … {% endif %}` | picks one branch (`elsif` is accepted for `elif`) | | `{% include 'name' %}` | renders another template found by `ITemplateIncludes`; its text is inserted where the tag stands | | `{% raw %} … {% endraw %}` | prints the delimiters inside as text | | `{# … #}` | a comment | | `{% image expr name=value … %}`, `{% html expr … %}`, `{% markdown expr … %}` | parsed and handed to the binder as a piece with the evaluated value and the options; the engine inserts nothing itself | | `scope=table-row` on `for` or `if` | a repetition-scope hint for the binder (see below) | * **Whitespace control.** `{{-`, `-}}`, `{%-` and `-%}` remove the white space next to the tag, inside the same paragraph or line. * **Tag-only units.** A paragraph or line that holds only block tags (and white space) produces nothing, so `{% for %}` and `{% endfor %}` on lines of their own leave no empty lines. A unit with an output tag or any text is kept, and an empty line stays an empty line. * **Delimiters** are configurable per template (`TemplateOptions.Delimiters`); the three openers must differ and none may begin another. * **Quotes.** Word's autocorrect turns `"` into curly quotes; `“ ” „ ‟` are read as `"` and `‘ ’ ‚ ‛` as `'` inside tags. A closing delimiter inside a string literal (`{{ "}}" }}`) does not end the tag. In `include`, `'name'` is a string. * **Loop variables.** `loop.index` (from 1), `loop.index0`, `loop.first`, `loop.last`, `loop.length`, `loop.revindex`, `loop.revindex0`, and `loop.parent` for the enclosing loop. * **Truthiness** (`if`): a boolean is itself; a number is true unless 0; text is true unless empty; a record is true; a collection is true when it has items; missing data and null are false. A formula that returns an array is true when any element is. * **Output.** Numbers print as the General format does (15 significant digits); TRUE/FALSE; dates from data print as ISO 8601 (`2024-03-09`, with the time when it is not midnight) and are serial numbers inside formulas, so `TEXT(d, "dd/mm/yyyy")` formats them; an array prints its cells separated by `, `; a record or null prints empty. Output is deterministic: `NOW()`, `TODAY()` and `RAND()` read `TemplateOptions.Clock` and `RandomSeed`. ## Expressions: Excel formulas plus two extensions Expressions are Excel formulas parsed by the calc parser and evaluated by the calc evaluator with the full function library, so `TEXT`, `SUM`, `IF`, `LET`, `LAMBDA`, `FILTER`, `SEQUENCE` and the number-format language behave as in a cell. Functions registered with `FunctionRegistry.AddCustom` (`TemplateOptions.Functions`) are the only extension point; the engine compiles no code and uses no reflection. These are **our extensions to Excel formula syntax**, not Excel behaviour: 1. **Dotted member paths.** `customer.name`, `items.price`: the first segment is a variable (a loop variable, or a member of the root data); later segments are members, matched ignoring case (an exact spelling wins). In the formula AST this is `MemberPathNode`; it prints back exactly as written. A path through `null` is `null`; a missing member is missing data. 2. **Array lifting.** A member of a collection is the collection of that member over its items: `items.price` is the array of every item's `price`, so operators and functions lift over it: `SUM(items.price * items.qty)`. A member that is itself a collection is flattened one level (`orders.lines.price`). A collection also has `length` and `count` when its items have no member of that name. An empty collection reads as one blank cell, so `SUM` gives 0 (Excel's empty array is `#CALC!`). A plain name or dotted path is read straight from the data, never parsed as a cell reference, so a variable called `rev7` works. Inside a larger formula a cell-style name such as `rev7` also reads the variable of that name (there is no grid); a range over such names is `#REF!`. A bare record used as a value (`{{ customer }}`) is `#VALUE!` in a formula and prints empty on its own. ## Data The engine reads data only through `ITemplateData` (null, scalar, record, sequence). `TemplateData.From(object)` adapts, without reflection: | Source | Notes | |---|---| | Classes marked `[TemplateModel]` (partial; also structs, records) | the source generator implements `ITemplateObject`: every public readable instance property and field, matched exactly then ignoring case | | `JsonDocument`, `JsonElement` | objects, arrays, numbers, strings, booleans, null | | `IDictionary` / `Dictionary` (any value type) | keys are read as text | | Any `IEnumerable` (lists, arrays) | a sequence; capped at `MaxRows` | | `DataTable`, `DataRow`, `DataRowView`, `IDataReader` | a table is a sequence of records by column name; a reader is read to the end (capped); `DBNull` is null | | XML (`TemplateData.FromXml`) | read with the library's tokenizer (no DTDs). Attributes and child elements are members; repeated elements are a sequence; text that reads as a number is a number in formulas and still prints as written | | strings, numbers, `bool`, dates, `Guid`, enums (their name), `byte[]` | scalars | Anything else (a class with no generated model) throws `OfficeUnsupportedException` rather than being read by reflection. ## Missing data and errors All problems with the template are found by `TemplateEngine.Parse`, before any data is bound: `ParsedTemplate.Defects` lists every error with its place. Rendering an invalid template throws `TemplateException` for the first. At render time the default is lenient: missing data renders empty and is a `template.missing-data` warning at the tag; an expression that evaluates to an error value prints that value and is `template.expression-error`. With `TemplateOptions.Strict`, the first of these throws `TemplateException`. Missing data that a formula handles (`IFERROR`, `ISBLANK`...) is not reported. ## Limits and sandbox * No file, network or clock access except through `ITemplateIncludes`, a supplied `TimeProvider` and registered functions. `include` is denied unless the caller supplies a resolver. * `TemplateOptions.MaxIterations` (1,000,000 loop iterations per render), `MaxOutputCharacters` (64M), `MaxBlockDepth` (64: nested blocks and includes together, also bounded by `OfficeLimits.MaxRecursionDepth`); `OfficeLimits` formula length, depth and evaluation limits apply to every expression. Exceeding one throws `OfficeLimitException`. * `CancellationToken` is checked on every node and loop iteration. ## The binder contract A binder converts a document part to a `TemplateSource`, renders it, and rebuilds the part. * **Units and segments.** A `TemplateUnit` is the structural thing that can be repeated or dropped (a Word paragraph, a table-cell paragraph, a slide text shape, a cell). It holds `TemplateSegment`s: the text of its runs, in order. Tags may be split across segments (Word splits a tag whenever it was typed in several edits, or spell-checked, or revised); the engine reads a unit's segments as one text. A tag never spans two units. * **Result.** `TemplateResult.Units` is the output: a list of `TemplateRenderedUnit(SourceUnit, Pieces)`. A source unit may appear several times (a loop body), once, or not at all (false branch, tag-only unit). A binder emits, for each, a clone of the source unit with its runs rebuilt from the pieces. * **Pieces.** `TemplatePiece(Kind, Segment, Text, Insertion)`: a text piece takes the formatting of segment `Segment` of the source unit. Text kept from the template stays in the segment it came from. A printed tag takes the segment where the tag *began*; a tag split over segments 3 and 4 prints into 3, and what follows the tag in 4 stays in 4. Adjacent text pieces from the same segment are merged. Image, HTML and Markdown tags come as non-text pieces with a `TemplateInsertion(Value, Options)` for the binder to render. * **Repetition scope.** By default a block repeats what lies between its tags in the unit order: inside one unit that is inline text (no new unit per iteration); across units it is the units between the tags, and each iteration starts a new output unit. To repeat larger structures (a table row whose tags sit in two cells, a slide, a sheet row) the binder implements `ITemplateScopeResolver` and returns the range of units a block covers, given the author's `scope=` hint and the units of the two tags. The engine then repeats (or keeps) those whole units, every iteration producing new units even when both tags sit in one unit, and the tags print nothing. A range that misses a tag is `template.scope-invalid`. * **Plain text.** `TemplateSource.FromText` makes one unit per line; `TemplateEngine.RenderText` is the plain-text binder. * **Locations.** `TemplateUnit.Location` (for example a part URI plus a path) is used for defects together with the offset in the unit's text; without it, defects carry a character offset in the whole text. ## Not here The DOCX and XLSX binders, Word mail merge, content-control binding, rendering of image/HTML/Markdown tags, and the `Template.Render` one-liner come with the binders. The presentation binder is described in `templating-pptx.md`. # Templating: word-processing documents Namespace `RecursiveAI.OneDocument.Documents.Templating` in the documents package. The engine, syntax, data model and defect codes are in `templating.md`; this page states how the document binder reads a document, what the blocks repeat, and what the data-fed tags insert. `Template.Render(templatePath, data, outputPath)` opens the file, finds the binder by its sniffed format, renders in place and saves in the format of the output extension. `Template.Render(Stream, data, Stream, OfficeFormat?)` does the same over streams, `Template.Render(OfficeFile, data)` renders an open file, and `Template.Validate(path)` returns the located errors without data. The CLI is `onedocument template