# onecontact docs, all pages (OneContact 1.0.0) Every page of https://recursiveai.org/onecontact/docs/ as markdown, getting started first. Index: https://recursiveai.org/onecontact/llms.txt # Getting started with OneContact Install OneContact 1.0.0 for .NET 10, then validate and format a phone number, check an email address, parse a web address, read a postal address and pull the contacts out of a small web page. Every sample on this page was run on the 1.0.0 packages from NuGet and shows its real output. The inputs are made up: phone numbers from ranges set aside for fiction, and `example.com` addresses. ## Install The packages target `net10.0`, so you need the .NET 10 SDK. ```shell dotnet new console -o OneContactStart cd OneContactStart dotnet add package RecursiveAI.OneContact --version 1.0.0 dotnet add package RecursiveAI.OneContact.Extraction --version 1.0.0 ``` Three packages are on NuGet at 1.0.0, all on one version: - **`RecursiveAI.OneContact`**: phones, email, web addresses, postal addresses, names and companies. It depends on the .NET base class library only. - **`RecursiveAI.OneContact.Extraction`**: `ContactExtractor`, which finds people, organizations and their contact fields in HTML and plain text. It brings in the core and OneHtml 1.6.0. - **`RecursiveAI.OneContact.Phones.Geo`**: the geocoder, carrier and time-zone mappers for phone numbers. None of them uses native code, and in the README’s words, “Nothing in the packages opens a socket, resolves a name or downloads a file.” Version 1.0 has no trained models. In the README’s words: “Version 1.0 runs on rules and markup alone. The trained model packages (`RecursiveAI.OneContact.Models.Tier1`, `.Models` and `.Models.Accurate`) follow in 1.1, after OneNLP 1.0.” Text the rules can’t read is refused with a reason, never guessed. The samples below are one `Program.cs`, run top to bottom. It starts with these lines: ```csharp using RecursiveAI.OneContact; using RecursiveAI.OneContact.Extraction; using RecursiveAI.OneContact.Extraction.Records; ``` ## Validate and format a phone number A national number needs a region; one written with `+` and its country code doesn’t. A parsed number has its E.164 form, every format, a number type and a validity check against the numbering plan. ```csharp // A national number needs a region; an international one does not. PhoneNumber phone = PhoneNumber.Parse("(202) 555-0145", Region.US); Console.WriteLine(phone.E164); Console.WriteLine(phone.Format(PhoneFormat.International)); Console.WriteLine(phone.Format(PhoneFormat.National)); Console.WriteLine(phone.Format(PhoneFormat.Rfc3966)); Console.WriteLine($"{phone.Type} | valid={phone.IsValid} | region {phone.Region}"); PhoneNumber london = PhoneNumber.Parse("+44 20 7946 0958"); Console.WriteLine($"{london.Format(PhoneFormat.National)} | {london.Type} | valid={london.IsValid} | region {london.Region}"); Console.WriteLine(PhoneNumber.TryParse("2025550145", out _) ? "parsed" : "refused: a national number needs a region"); Console.WriteLine(Contacts.SameKey("(202) 555-0145", "+1 202.555.0145", Region.US)); ``` ```text +12025550145 +1 202-555-0145 (202) 555-0145 tel:+1-202-555-0145 FixedLineOrMobile | valid=True | region US 020 7946 0958 | FixedLine | valid=True | region GB refused: a national number needs a region True ``` A national number with no region is refused rather than guessed. `Contacts.SameKey` compares two values by their comparison key, so differently written forms of one number match. The project checks these results against libphonenumber. For parsing, validation, number types, every format, region, possibility and example numbers, across every region and non-geographic code, the README counts mismatches on generated cases against libphonenumber 9.0.43: “0 of 2,819,630 results (141,206 cases, 2 seeds)”. [Phone numbers](https://recursiveai.org/onecontact/docs/phones/) covers the as-you-type formatter, the matcher that finds numbers in text, and short numbers. ## Check an email address `EmailAddress.Parse` reads one mailbox by RFC 5322 and RFC 6531, with or without a display name, and throws `ContactParseException` with a reason when the address isn’t one. `TryParse` does the same without the exception. ```csharp foreach (string text in new[] { "Jo Example ", "sales@example.com", "jo@example..com" }) { try { EmailAddress email = EmailAddress.Parse(text); Console.WriteLine($"{email.Canonical} | {email.DisplayName} | {email.Flags}"); } catch (ContactParseException e) { Console.WriteLine($"refused: {e.Reason.Id}"); } } ``` ```text Jo.Example+news@example.com | Jo Example | HasDisplayName sales@example.com | | RoleAccount refused: email.invalid-domain ``` The canonical form keeps the local part as written and lowercases the domain. `RoleAccount` marks a role address such as `sales@`. The empty label in `example..com` makes the domain invalid. Flags such as `FreeProvider`, `Disposable` and `PossibleTypo` only report: in the docs’ words on typo suggestions, “nothing is ever changed”. [Email addresses](https://recursiveai.org/onecontact/docs/email/) has every rule, flag and provider key. ## Parse a web address `WebAddress` follows the WHATWG URL Standard and reads the registrable domain from the Public Suffix List. The canonical form drops tracking parameters and says so in its flags. ```csharp WebAddress web = WebAddress.Parse("HTTPS://Shop.Example.co.uk:443/a/../Sale?utm_campaign=fall&id=7"); Console.WriteLine(web.Canonical); Console.WriteLine($"{web.Host} | {web.RegistrableDomain} | {web.Flags}"); Console.WriteLine($"key {web.Key}"); ``` ```text https://shop.example.co.uk/Sale?id=7 shop.example.co.uk | example.co.uk | TrackingParametersRemoved key example.co.uk/sale?id=7 ``` The scheme and host are lowercased, the default port and the `..` segment are gone, and `utm_campaign` is removed while `id` stays. `co.uk` is a public suffix, so the registrable domain is `example.co.uk`. [Web addresses](https://recursiveai.org/onecontact/docs/web/) lists the tracking parameters that are removed and the social-profile flags. ## Read a postal address `PostalAddress.Parse` reads free text, writes it back in the postal authority’s style, and validates it against the country’s structure. Full free-text parsing covers the United States, Canada, Mexico, Australia, the United Kingdom and Crown dependencies, and the Latin-script countries of Europe; every other country has the structural layer (fields, layout, required fields, postcode patterns). ```csharp PostalAddress address = PostalAddress.Parse("Jo Example\n123 north main street suite 400\nspringfield, il 62704"); Console.WriteLine($"{address.Country.Code} | {address.Locality} | {address.AdminArea} | {address.PostalCode} | {address.Coverage}"); Console.WriteLine(address.ToString(AddressStyle.Postal, AddressLayout.Label)); Console.WriteLine(address.ToString(AddressStyle.Display, AddressLayout.SingleLine)); Console.WriteLine($"{address.Validate().Count} structural issues"); ``` ```text US | springfield | IL | 62704 | Full JO EXAMPLE 123 N MAIN ST STE 400 SPRINGFIELD IL 62704 Jo Example, 123 N Main St Ste 400, Springfield, IL 62704 0 structural issues ``` The country came from the text: a state and a ZIP code. The fields keep what was written (`springfield`), and the two styles standardize it: `Postal` is USPS Publication 28 (capitals, `N`, `ST`, `STE`), and `Display` is the same words in title case. When the text shows its country, the country is read from it and flagged `CountryInferred`. When it doesn’t, `PostalAddress` refuses it, and you give the country: ```csharp // The country is read from the text when the text shows it ... PostalAddress inferred = PostalAddress.Parse("Musterstraße 12, 10115 Berlin"); Console.WriteLine($"{inferred.Country.Code} | {inferred.Flags} | {inferred.ToString(AddressStyle.Postal, AddressLayout.SingleLine)}"); // ... and text that doesn't say which country it is in is refused: give the country. string market = "Am Markt 5, 10117 Berlin"; Console.WriteLine(PostalAddress.TryParse(market, out _) ? "parsed" : "refused: the country can't be told"); Console.WriteLine(PostalAddress.Parse(market, Region.Parse("DE")).ToString(AddressStyle.Postal, AddressLayout.SingleLine)); ``` ```text DE | CountryInferred | Musterstraße 12, 10115 BERLIN refused: the country can't be told Am Markt 5, 10117 BERLIN ``` In the README’s words, the rules “never read a text in the wrong country and never print a value the text does not contain.” [When the country can’t be told](https://recursiveai.org/onecontact/docs/address-country-undetermined/) explains the inference, and each country group has its own page under Postal addresses. ## Extract contacts from a web page `ContactExtractor` reads a page you give it: structured markup (JSON-LD, microdata, RDFa Lite, h-card, `mailto:` and `tel:` links, `
`) and the visible text. It groups what it finds into person and organization records. It never fetches anything. ```csharp string html = """ Contact · Example Bakery

Contact us

Wholesale orders

Jo Example, Head Baker

Phone: (217) 555-0188

Email: jo [at] example [dot] com

"""; ExtractResult result = new ContactExtractor().ExtractHtml(html); foreach (ContactRecord r in result.Records) { Console.WriteLine($"{r.Kind}: {r.Name?.Text}"); foreach (ExtractedField f in r.Fields) Console.WriteLine($" {f.Kind,-8} {f.Text.Replace("\n", ", "),-42} {f.Source} {f.Flags}"); } ``` ```text Organization: Example Bakery Name Example Bakery JsonLd None Phone +1 217 555 0123 JsonLd None Website https://bakery.example.com/ JsonLd None Address 10 Elm Street, Springfield, IL, 62704, US JsonLd Composed Person: Jo Example Name Jo Example Text None Phone (217) 555-0188 Text RegionInferred Email jo@example.com Text Deobfuscated ``` The bakery comes from its JSON-LD, and its address is `Composed` from the structured parts. Jo Example comes from the visible text. The national phone number took its region from the page’s `lang="en-US"` (`RegionInferred`), and the `[at]`/`[dot]` email was decoded (`Deobfuscated`). In the docs’ words, “Job titles are evidence only and are never returned.” Every field also carries its offsets into the page, a confidence and its normalized value. `ExtractText` does the same for plain text such as a CRM note. Names, emails, phones and addresses are personal data. In the docs’ words, extraction “runs offline on exactly what the caller passes in, and the library keeps no copy; responsibility for a lawful basis and for compliance with data-protection law (for example the GDPR) for what is extracted and what is done with it sits with the caller.” See [extraction](https://recursiveai.org/onecontact/docs/extraction/). ## Next - [Overview](https://recursiveai.org/onecontact/docs/overview/): the packages, the quick start and every result with what it was measured against. - [Coming from another library](https://recursiveai.org/onecontact/docs/migrating/): libphonenumber, libpostal, python-nameparser and probablepeople, call by call. - [Person names](https://recursiveai.org/onecontact/docs/names/) and [company names](https://recursiveai.org/onecontact/docs/company-names/): titles, particles, name order, several people in one string, and ISO 20275 legal forms. - [Geocoder, carrier and time zones](https://recursiveai.org/onecontact/docs/phones-geo/): `RecursiveAI.OneContact.Phones.Geo`, with the original-carrier caveat. # OneContact libphonenumber- and libpostal-class contact data for .NET. A clean-room library for .NET 10 with no native code and no network, NativeAOT and trim clean. It turns one value of a known kind (a phone number, an email address, a web address, a postal address, a person's name, a company name) into its canonical form, a versioned comparison key, flags, and offsets back into the input. A second package finds those values in web pages and free text and groups them into person and organization records. - **Phones** for every region: parse, validate, number type, every format, an as-you-type formatter, a matcher with leniency levels, short numbers, example numbers. Geocoder, carrier and time-zone mappers in an add-on. - **Email** (RFC 5322, RFC 6531, IDNA domains) with provider-aware keys and role, free-provider and disposable flags. - **Web addresses**: WHATWG URL and the Public Suffix List, tracking parameters removed. - **Addresses**: a structural layer for every country (fields, layout, required fields, postcode patterns), and full free-text parsing, standardization to the national postal authority's style, validation and expansion for the United States, Canada, Mexico, Australia, the United Kingdom and Crown dependencies, the Latin-script countries of the EU, Switzerland, Norway, Iceland, Liechtenstein and the small European states. - **Names**: Western-pattern person names (titles, suffixes, particles, order, several people in one string, casing restoration) and company names with ISO 20275 legal forms. - **Extraction**: people, organizations, addresses, phones, emails and websites from pages (JSON-LD, microdata, RDFa Lite, h-card, `mailto:`/`tel:`, `
`, visible text) and from plain text. Everything runs offline. Nothing in the packages opens a socket, resolves a name or downloads a file. The library keeps no state about what you parse, and `CultureInfo` never changes a result. ## Packages All packages share one version. | Package | Depends on | Contents | |---|---|---| | `RecursiveAI.OneContact` | the .NET base class library only | The value types, `Contacts`, keys and flags, JSON converters; phones, email, web addresses, addresses (structural layer for every country; rules for the Tier 1 and Tier 2 countries), names and companies | | `RecursiveAI.OneContact.Phones.Geo` | core | Geocoder, carrier and time-zone mappers for phone numbers | | `RecursiveAI.OneContact.Extraction` | core, OneHtml | `ContactExtractor`: markup readers, visible-text reading, obfuscation decoding, records, site merge | Version 1.0 runs on rules and markup alone. The trained model packages (`RecursiveAI.OneContact.Models.Tier1`, `.Models` and `.Models.Accurate`) follow in 1.1, after OneNLP 1.0. The rules read what is certain (phone metadata, postcode patterns, region names, authority abbreviations, legal forms, titles, line structure) and say so in `ParseDetail`. They never read a text in the wrong country and never print a value the text does not contain. Text they cannot read is refused with a reason: when only a trained model could fill a field, the reason names the package that does so, available from 1.1; when the country cannot be told, `address.country-undetermined` asks you to give one. ## Quick start ``` dotnet add package RecursiveAI.OneContact ``` The program below is [samples/OneContact.QuickStart](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/samples/OneContact.QuickStart/Program.cs), built with the solution and run before each release, so what you read here compiles. ```csharp using RecursiveAI.OneContact; // Phones: national input needs a region; international input does not. PhoneNumber phone = PhoneNumber.Parse("(202) 456-1111", Region.US); Console.WriteLine($"{phone.E164} | {phone.Format(PhoneFormat.International)} | {phone.Format(PhoneFormat.National)} | {phone.Type} | valid={phone.IsValid}"); Console.WriteLine(PhoneNumber.TryParse("2024561111", out _) ? "parsed" : "a national number without a region is refused"); // Email: the canonical form keeps what the standard keeps; the key is the aggressive, versioned form. EmailAddress email = EmailAddress.Parse("\"J. Smith\" "); Console.WriteLine($"{email.Canonical} | key {email.Key} | {email.Flags}"); // Web addresses: WHATWG URL, IDNA and the Public Suffix List. WebAddress web = WebAddress.Parse("www.Example.com/?utm_source=x"); Console.WriteLine($"{web.Canonical} | {web.RegistrableDomain} | key {web.Key} | {web.Flags}"); // Addresses: parse free text, write it back in the postal authority's style, validate its structure. PostalAddress address = PostalAddress.Parse("1 main street apt 4, springfield, va 22162"); Console.WriteLine($"{address.Country.Code} | {address.Locality} | {address.PostalCode} | {address.Coverage}"); Console.WriteLine(address.ToString(AddressStyle.Postal, AddressLayout.SingleLine)); Console.WriteLine(address.ToString(AddressStyle.Display, AddressLayout.SingleLine)); Console.WriteLine($"{address.Validate().Count} structural issues"); // Text that does not say which country it is in is refused; give the country. Console.WriteLine(PostalAddress.TryParse("Unter den Linden 77, 10117 Berlin", out _) ? "parsed" : "the country cannot be told: give one"); Console.WriteLine(PostalAddress.Parse("Unter den Linden 77, 10117 Berlin", Region.Parse("DE")).Country.Code); // Names and companies. PersonName person = PersonName.Parse("DR. MARÍA JOSÉ GARCÍA LÓPEZ", Region.Parse("ES")); Console.WriteLine($"{person.Prefix.Text} | {person.Given.Text} | {person.Family.Text} | {person.SecondFamily.Text} | key {person.Key}"); CompanyName company = CompanyName.Parse("Muster Aktiengesellschaft", Region.Parse("DE")); Console.WriteLine($"{company.Canonical} | {company.Base.Text} | {company.LegalForm?.ElfCode}"); // The same key means the same thing, whatever the kind. Console.WriteLine(Contacts.Normalize("j.smith@example.com").Kind); Console.WriteLine(Contacts.SameKey("(202) 456-1111", "+1 202-456-1111", Region.US)); ``` It prints: ``` +12024561111 | +1 202-456-1111 | (202) 456-1111 | FixedLineOrMobile | valid=True a national number without a region is refused J.Smith+crm@googlemail.com | key jsmith@gmail.com | FreeProvider, HasDisplayName https://www.example.com/ | example.com | key example.com | SchemeAssumed, TrackingParametersRemoved US | springfield | 22162 | Full 1 MAIN ST APT 4, SPRINGFIELD VA 22162 1 Main St Apt 4, Springfield, VA 22162 0 structural issues the country cannot be told: give one DE Dr. | María | García | López | key maria j garcialopez Muster AG | Muster | 6QQB Email True ``` The sample also shows the matcher (`PhoneNumber.FindAll`), the geocoder from `.Phones.Geo`, JSON through the source-generated `ContactJsonContext`, and extraction: ```csharp ExtractResult note = new ContactExtractor().ExtractText("Name: Jo Smoke\nTel: +1 202 555 0145\njo@example.com"); foreach (ContactRecord r in note.Records) { Console.WriteLine($"{r.Kind}: {r.Name?.Text} | {r.Phones[0].Text} | {r.Emails[0].Text}"); } // Person: Jo Smoke | +1 202 555 0145 | jo@example.com ``` For pages, `ContactExtractor.Extract(html)` returns the records and `ExtractHtml` returns every field with its source, confidence and byte offsets into the original HTML. Pages of one site merge with `ExtractSite`. Disagreeing values (a stale phone in markup, a different one in the text) are kept, each with its source. Extraction runs on what you give it and stores nothing; responsibility for lawful processing of the personal data it finds sits with the caller (see [docs/extraction.md](https://recursiveai.org/onecontact/docs/extraction/)). Low-level forms take spans and return `OperationStatus` plus a `ParseDetail`; phones, names and emails also have batch forms (`PhoneNumberParser.ParseMany`, `NameParsing.SplitMany`, `EmailParser.ParseMany`), and addresses are parsed one at a time. The low-level address form keeps a partial reading when the country cannot be told (`AddressFacts.CountryUndetermined`), where `PostalAddress.Parse` refuses. Explicit metadata (`PhoneMetadata.LoadXml`; numbers parsed with it answer from it) can be supplied instead of the built-in tables. ## Coming from another library [docs/migrating.md](https://recursiveai.org/onecontact/docs/migrating/) maps libphonenumber (Java and C#), libpostal (`parse_address`, `expand_address`) and python-nameparser / probablepeople calls to OneContact. It is a table of what to call, not a compatibility layer. ## Offline, NativeAOT and trimming - Core references the .NET base class library only. No native code, no P/Invoke, no reflection-based serialization, no `Reflection.Emit`. - Metadata, structural tables, authority rules, dictionaries and legal-form tables are compiled into the assembly as data and read on first use. - `IsAotCompatible` and `IsTrimmable` are on, with warnings as errors. [samples/OneContact.AotSmoke](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/samples/OneContact.AotSmoke/Program.cs) is published with NativeAOT and exercises the library. - JSON uses `Utf8JsonReader`/`Utf8JsonWriter` and a source-generated context (`ContactJsonContext`). ## Results The library is measured against the leaders it replaces. Every number below was measured; the evaluation sets behind real-text rows are private and only aggregates are published. ### Exact agreement | Part | Scope | Metric | OneContact | Reference | Verdict | |---|---|---|---|---|---| | Phones: parse, validate, type, every format, region, possibility, examples | every region and non-geographic code | mismatches on generated cases | 0 of 2,819,630 results (141,206 cases, 2 seeds) | libphonenumber 9.0.43 | exact | | Phones: as-you-type, matcher (every leniency), short numbers | every region | mismatches on generated cases | 0 of 969,881 results (207,679 cases, 2 seeds) | libphonenumber 9.0.43 | exact | | Phone geocoder, carrier and time zones (`Phones.Geo`) | every calling code | mismatches on generated cases | 0 of 1,159,521 cases (2 seeds) | libphonenumber geocoder 3.43, carrier 2.43 | exact | | URL | WPT `urltestdata.json` | entries passing | 896 / 896 | WHATWG URL Standard conformance data | exact | | URL | WPT `setters_tests.json`, `toascii.json` | entries passing | 278 / 278, 87 / 87 | same | exact | | Internationalized domain names | `IdnaTestV2.json` | entries passing | 2676 / 2676 | UTS #46 conformance data | exact | | Public Suffix List | `test_psl.txt` | tests passing | 78 / 78 | the list's own test file | exact | | Email | our RFC 5322 / 6531 corpus | cases passing | 205 / 205 | the RFCs' examples | exact | | Email | public isemail test list | cases passing | 164 / 164, with 35 documented expectation overrides | the list's expectations | exact | | Address standardization, United States and Canada | the authorities' published examples | examples reproduced | 206 / 206 | USPS Publication 28, Canada Post (EN, FR) | exact | | Address standardization, Tier 2 countries | the posts' and UPU's published examples, five example sets | examples reproduced | 316 / 316 (five example sets of 57, 54, 51, 65 and 89) | national postal authorities, UPU addressing sheets | exact | | Structural address layer | every libaddressinput region, generated cases | mismatches | 0 of 32,883, with 4 documented intended differences | libaddressinput (released build) | exact, with the 4 differences | Fuzzing: every parser and loader the packages ship (20 targets: phone strings, the matcher, as-you-type, phone and short-number metadata, email, web addresses, IDNA, addresses, names, companies, `Contacts.Normalize`, HTML and text extraction, the model-file loader that ships with the 1.1 packages, JSON converters and the compiled-table reader) was fuzzed for 600 seconds each with AFL++ and SharpFuzz. One bug turned up (a negative span length in the address reader) and is fixed with a regression test; the rerun of that target found none. See [fuzz/README.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/fuzz/README.md). The four intended differences of the structural layer: (1) we ship a newer data snapshot than the reference build's built-in table; (2) a country with no separate Latin layout uses its main layout, where the reference uses a default one; (3) any language tag with a `Latn` script subtag selects the Latin layout; (4) the convenience `Create` lowercases tags and maps `_` to `-`. ### Honest refusal The rules never read a text in the wrong country and never print a value the text does not contain; what they cannot read they refuse with a reason. Measured on 4,623 items of real pages (the United States, Canada, every Tier 2 country), with and without the country given: with it, 0 readings in a wrong country; without it, 1 of 3,098 readings with a country was in a wrong country (a Czech-labelled item whose text is Slovak) and 925 more were returned without a country rather than guessed; one standardized state abbreviation (`D.F.` written `CDMX`) was the only printed word not found in its input. Every cause is listed in [docs/maintainers/refusal-rows.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers/refusal-rows.md). ### Speed Median time, ours divided by the comparator's (below 1 is faster), measured with BenchmarkDotNet on an x64 machine (AMD Ryzen 9 7950X3D) and an arm64 machine (Apple M2), each alone on the machine. Times, allocations and peak memory per row: [docs/maintainers/speed-gate.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers/speed-gate.md). | Operation | Comparator | x64 | arm64 | Verdict | |---|---|---|---|---| | Phone parse | libphonenumber-csharp 9.0.42 | 0.71 | 0.75 | faster | | Phone parse and validate | same | 0.27 | 0.31 | faster | | Phone format | same | 0.23 | 0.23 | faster | | Phone number type | same | < 0.01 | < 0.01 | faster | | As-you-type formatter | same | 0.51 | 0.49 | faster | | Find numbers in text (Possible / Valid / StrictGrouping / ExactGrouping) | same | 0.87 / 0.53 / 0.54 / 0.53 | 0.97 / 0.60 / 0.60 / 0.59 | faster | | Short numbers (validity and cost / emergency) | same | 0.34 / 0.67 | 0.26 / 0.66 | faster | | Geocoder / carrier / time zones | same | 0.80 / 0.53 / 0.59 | 0.83 / 0.53 / 0.59 | faster | | Email parse (mixed / ASCII / IDN) | MimeKit 4.18.2 `MailboxAddress` | 0.84 / 0.86 / 0.82 | 0.81 / 0.80 / 0.94 | faster | | Email parse (mixed / ASCII / IDN) | `System.Net.Mail.MailAddress` | 1.98 / 1.85 / 2.82 | 1.50 / 1.43 / 2.45 | slower; more accurate (agrees with the RFC verdict on 32 of 33 cases, against 30) | | URL parse | AngleSharp 1.9.1 `Url` | 1.01 | 0.94 | at par; more accurate (WPT 896 / 896, against 503 / 896) | | Public suffix lookup | Nager.PublicSuffix 3.8.0 | 0.24 | 0.29 | faster | | Person name parse | NameParserSharp 1.5.0 | 0.70 | 0.73 | faster | | US address parse, country given | AddressParser 1.0.3 (US-only regex parser) | 0.10 | 0.10 | faster and more accurate | | Extraction from text | Microsoft.Recognizers.Text 1.8.13 sequence finding | 0.51 | 0.34 | faster | | Extraction from HTML | same | 0.77 | 0.50 | faster | The US-only comparator assumes a US address, so the address row gives OneContact the country too; without it, OneContact first works out which country the text is in (0.96 on x64, 1.05 on arm64). First call in a fresh process: 13 to 75 ms per kind, 359 ms for HTML extraction (x64). ### Size | Package | Size | libpostal | |---|---|---| | `RecursiveAI.OneContact` | 1.39 MB | about 2 GB of data | | `RecursiveAI.OneContact.Phones.Geo` | 1.52 MB | | | `RecursiveAI.OneContact.Extraction` | 0.21 MB | | Trained-model quality rows (address parsing, names, legal forms and extraction against libpostal, probablepeople, python-nameparser, cleanco and spaCy) are published with the model packages in 1.1. How the rows are measured is described in [docs/index.md](https://recursiveai.org/onecontact/docs/docs-map/) under evaluation. ## Building ``` dotnet build OneContact.Release.slnx -c Release dotnet test --solution OneContact.Release.slnx -c Release ``` `OneContact.Release.slnx` holds the packages, their tests and samples, and restores from nuget.org; `OneContact.slnx` adds the model packages, tools and benchmarks, which need preview packages that are not on nuget.org yet. Build output goes outside the source tree: set `ONECONTACT_ARTIFACTS` to a directory, or leave it unset to use `artifacts/`. How the code and its data fit together: [docs/architecture.md](https://recursiveai.org/onecontact/docs/architecture/). Where the tools keep data, models and evaluation sets: [docs/tools-environment.md](https://recursiveai.org/onecontact/docs/tools-environment/). All documentation: [docs/index.md](https://recursiveai.org/onecontact/docs/docs-map/). Licensed `MIT OR Apache-2.0`. Copyright (c) Recursive AI, LLC. Third-party data is listed in [THIRD-PARTY-NOTICES.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/THIRD-PARTY-NOTICES.md). Home: # OneContact documentation Start with the [README](https://recursiveai.org/onecontact/docs/overview/) for the packages, the quick start and the results. Documents about the trained model packages are marked *from 1.1*. Coming from another library? Read [migrating.md](https://recursiveai.org/onecontact/docs/migrating/) first. The rest of this folder is ordered the way you will probably need it. Documents marked *maintainer* describe how the project is built and measured, not how to use it. ## Using the library ### Phones - [phones.md](https://recursiveai.org/onecontact/docs/phones/): parsing, validation, number types, every format, the as-you-type formatter, the matcher and its leniency levels, short numbers, example numbers, the metadata and how to load newer metadata. - [phones-geo.md](https://recursiveai.org/onecontact/docs/phones-geo/): the geocoder, carrier and time-zone mappers (`RecursiveAI.OneContact.Phones.Geo`), with the original-carrier caveat. ### Email - [email.md](https://recursiveai.org/onecontact/docs/email/): what `EmailAddress` reads (RFC 5322, RFC 6531), the canonical form, flags (role account, free provider, disposable), the key rules per provider, typo suggestions. ### Web addresses - [web.md](https://recursiveai.org/onecontact/docs/web/): `WebAddress`, WHATWG URL parsing, canonical form, tracking parameters, the key, social-profile flags. - [psl.md](https://recursiveai.org/onecontact/docs/psl/): the Public Suffix List lookups and how our additions sit beside the unmodified list. - [idna.md](https://recursiveai.org/onecontact/docs/idna/): internationalized host names (UTS #46). ### Addresses - [address-structural.md](https://recursiveai.org/onecontact/docs/address-structural/): the worldwide structural layer: fields, layout, required fields, postcode patterns, `Validate()`. - [address-rules-us-ca.md](https://recursiveai.org/onecontact/docs/address-rules-us-ca/): United States and Canada: parsing and the postal authorities' styles (USPS Publication 28, Canada Post). - Tier 2 countries, one document per group of postal authorities: [address-rules-tier2-a.md](https://recursiveai.org/onecontact/docs/address-rules-tier2-a/) (FR, BE, NL, LU, DE, AT, CH, LI, MC), [address-rules-tier2-b.md](https://recursiveai.org/onecontact/docs/address-rules-tier2-b/) (GB and Crown dependencies, IE, ES, PT, IT, MT, SM, VA, AD), [address-rules-tier2-c-north.md](https://recursiveai.org/onecontact/docs/address-rules-tier2-c-north/) (DK, SE, FI, NO, IS, EE, LV, LT), [address-rules-tier2-c-south.md](https://recursiveai.org/onecontact/docs/address-rules-tier2-c-south/) (PL, CZ, SK, SI, HR, HU, RO, MX, AU). - [address-last-line-orders.md](https://recursiveai.org/onecontact/docs/address-last-line-orders/): how the last line (postcode, town, region) is read in each country's order. - [address-country-undetermined.md](https://recursiveai.org/onecontact/docs/address-country-undetermined/): inferring the country from the text, and what is reported when several fit. - [address-dictionaries.md](https://recursiveai.org/onecontact/docs/address-dictionaries/) and [address-keys.md](https://recursiveai.org/onecontact/docs/address-keys/): expansion dictionaries and the address key. - [api-shapes.md](https://recursiveai.org/onecontact/docs/api-shapes/): parse overloads, reasons, regions, formats, explicit models, JSON and limits. ### Names and companies - [names.md](https://recursiveai.org/onecontact/docs/names/): person names: hints, classes, parts, casing restoration, order, multi-person splits. - [name-keys.md](https://recursiveai.org/onecontact/docs/name-keys/): the person key. - [company-names.md](https://recursiveai.org/onecontact/docs/company-names/): company names, ISO 20275 legal forms with ELF codes, DBA and c/o, the company key. ### Extraction - [extraction.md](https://recursiveai.org/onecontact/docs/extraction/): `ContactExtractor`: structured markup, visible text, obfuscation decoding, records, site merge, disagreements, limits. It carries the data-protection note. ### Models (from 1.1) Version 1.0 runs on rules and markup alone. The model packages follow in 1.1, after OneNLP 1.0; these documents describe them and the seam in core that they plug into. - [models.md](https://recursiveai.org/onecontact/docs/models/): how a model package reaches the parsers, what the Tier 1 model is, how it was trained. - [models-superset.md](https://recursiveai.org/onecontact/docs/models-superset/): the `RecursiveAI.OneContact.Models` package (United States, Canada and the Tier 2 countries). - [models-accurate.md](https://recursiveai.org/onecontact/docs/models-accurate/): the `RecursiveAI.OneContact.Models.Accurate` package. ### Keys, data and notices - Keys are described next to each kind: [phones.md](https://recursiveai.org/onecontact/docs/phones/), [email.md](https://recursiveai.org/onecontact/docs/email/), [web.md](https://recursiveai.org/onecontact/docs/web/), [address-keys.md](https://recursiveai.org/onecontact/docs/address-keys/), [name-keys.md](https://recursiveai.org/onecontact/docs/name-keys/), [company-names.md](https://recursiveai.org/onecontact/docs/company-names/). Any change to key rules ships as a new `KeyVersion`; a published version never changes. - [THIRD-PARTY-NOTICES.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/THIRD-PARTY-NOTICES.md): the licences and credits of the data the packages carry. - `upstream//PIN`: the exact upstream release and file hashes behind each compiled data table. ## Maintainers and evaluation - [architecture.md](https://recursiveai.org/onecontact/docs/architecture/) *(maintainer)*: how code and compiled data fit together, the data file format, how value types are shaped. - [tools-environment.md](https://recursiveai.org/onecontact/docs/tools-environment/) *(maintainer)*: where the tools keep data, models and evaluation sets. - [maintainers/refusal-rows.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers/refusal-rows.md) *(maintainer)*: the honest-refusal rows, how wrong-country and invented readings are counted, and every cause found. - [maintainers/](https://github.com/recursive-ai-oss/onecontact-csharp/tree/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers) *(maintainer notes)*: measured rows and tuning history ([models-superset.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers/models-superset.md), [models-tier1-measurements.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers/models-tier1-measurements.md), [extraction-measurements.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers/extraction-measurements.md), [address-recovery.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers/address-recovery.md)), the address challenger models ([models-challenger.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers/models-challenger.md)), the crawl-labelled span head ([extraction-crawl-training.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers/extraction-crawl-training.md)), and how the real-text evaluation sets are built, labelled and scored ([evaluation-sets.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers/evaluation-sets.md), [extraction-eval-sets.md](https://github.com/recursive-ai-oss/onecontact-csharp/blob/90204cd135c124cf1cb0ed3d7b2d40bab55e645b/docs/maintainers/extraction-eval-sets.md)). The sets themselves are private; only aggregate numbers are published. - Training data *(maintainer)*: [training-data-address-us-ca.md](https://recursiveai.org/onecontact/docs/training-data-address-us-ca/), [training-data-address-tier2-a.md](https://recursiveai.org/onecontact/docs/training-data-address-tier2-a/), [training-data-address-tier2-b.md](https://recursiveai.org/onecontact/docs/training-data-address-tier2-b/), [training-data-address-tier2-c.md](https://recursiveai.org/onecontact/docs/training-data-address-tier2-c/), [training-data-address-challenger.md](https://recursiveai.org/onecontact/docs/training-data-address-challenger/): which open registers are rendered into training examples, under which licences. # Migrating to OneContact Three reads-like tables for people who know the libraries OneContact replaces. Each row names a call or concept from the other library's published documentation and the OneContact call that does the same job. There is no compatibility layer and no copied API shape: OneContact uses .NET conventions (value types, `Parse`/`TryParse`, enums, `OperationStatus`), so the names differ and a few ideas are folded together or left out on purpose. Where a feature has no equivalent, the row says so. The other libraries are described here only from their public documentation (their READMEs, Javadoc, wikis and user guides). If a behaviour matters to you, check it against your data: the phone and structural-address layers are tested for exact agreement with their reference (see the Results section of the [README](https://recursiveai.org/onecontact/docs/overview/)), the parsing layers are measured, not identical. Habits that change for everyone: - **No default region, no default culture.** A national phone number needs an explicit `Region`; text that is not an international number fails with `phone.region-required` rather than guessing. Nothing reads `CultureInfo`. - **Value types.** `PhoneNumber`, `EmailAddress`, `WebAddress`, `PostalAddress`, `PersonName` and `CompanyName` are immutable structs. `==` compares the canonical form; compare keys explicitly with `Contacts.SameKey` or `.Key`. - **Errors.** `Parse` throws `ContactParseException` (a `FormatException`) carrying a `ReasonCode`; `TryParse` returns `bool`; the low-level `TryParse(span, out value, out ParseDetail)` returns `OperationStatus` and a detail with issues, field offsets, confidences and whether a rule or a model decided each field. - **Offsets.** Every value and part carries a `SourceSpan` of UTF-16 offsets into the original input. - **Never invented.** OneContact does not fill in a city from a postcode, guess gender or origin, or apply a typo fix. Typos come back as suggestions (`EmailAddress.TypoSuggestion`), facts as flags. ## libphonenumber (Java) and libphonenumber-csharp → OneContact Both libraries expose one `PhoneNumberUtil` singleton with the same method names (the C# port in Pascal case) and a mutable `PhoneNumber` message. OneContact has an immutable `PhoneNumber` struct with the common questions as members, and a `PhoneNumberParser` engine class for everything that takes a metadata choice. Namespaces: `RecursiveAI.OneContact.Phones` for the engine, enums and matcher; `RecursiveAI.OneContact` for `PhoneNumber` and `Region`. Regions are a `Region` struct (`Region.US`, `Region.Parse("DE")`), not strings. | libphonenumber | OneContact | |---|---| | `PhoneNumberUtil.getInstance()` | `PhoneNumberParser.Default` (most calls are also members of `PhoneNumber`) | | `parse(text, region)` | `PhoneNumber.Parse(text, Region)`, `PhoneNumber.TryParse(text, Region, out value)` | | `parse` of an international number (`+...`), region `ZZ` or null | `PhoneNumber.Parse(text)` (no region) | | `parseAndKeepRawInput` | `PhoneNumberParser.TryParse(span, region, keepRawInput: true, out number, out detail)`; `number.RawInput`, `number.CountryCodeSource` | | `NumberParseException` and its `ErrorType` | `ContactParseException.Reason`: `PhoneRegionRequired`, `PhoneNotANumber`, `PhoneInvalidCountryCode`, `PhoneTooShortAfterIdd`, `PhoneTooShort`, `PhoneTooLong` (ids such as `phone.too-short`) | | `isValidNumber` | `number.IsValid` | | `isValidNumberForRegion` | `number.IsValidForRegion(region)` | | `isPossibleNumber` / `isPossibleNumberWithReason` | `number.GetPossibility()` returns `PhonePossibility` (`IsPossible`, `IsPossibleLocalOnly`, `TooShort`, `TooLong`, `InvalidLength`, `InvalidCountryCode`); `GetPossibility(type)` checks one number type | | `getNumberType` | `number.Type` (`PhoneNumberType`) | | `getRegionCodeForNumber` | `number.Region` | | `getRegionCodesForCountryCode`, `getCountryCodeForRegion` | `PhoneMetadata.Default.GetRegions(callingCode)`, `GetCallingCode(region)` | | `getSupportedRegions`, `getSupportedCallingCodes` | `PhoneMetadata.Default.Regions`, `.CallingCodes` | | `getSupportedTypesForRegion` | `PhoneNumberParser.Default.GetSupportedTypes(region)` | | `getNationalSignificantNumber` | `number.NationalSignificantNumber` | | `getCountryCode`, `getNationalNumber`, `hasExtension`, `getExtension` | `number.CountryCode`, `.NationalNumber`, `.Extension` (null when absent) | | `isItalianLeadingZero`, `getNumberOfLeadingZeros` | `number.ItalianLeadingZero`, `.LeadingZeroCount` | | `format(number, E164 / INTERNATIONAL / NATIONAL / RFC3966)` | `number.Format(PhoneFormat.E164 / International / National / Rfc3966)`; `number.E164` | | `formatOutOfCountryCallingNumber` | `number.FormatOutOfCountry(callingFrom)` | | `formatNumberForMobileDialing` | `number.FormatForMobileDialing(callingFrom, withFormatting)` | | `formatInOriginalFormat` | `number.FormatInOriginalFormat(callingFrom)` (returns the raw input where formatting would change a digit) | | `formatNationalNumberWithCarrierCode`, `...WithPreferredCarrierCode` | `PhoneNumberParser.Default.FormatNationalWithCarrierCode`, `FormatNationalWithPreferredCarrierCode` | | `formatOutOfCountryKeepingAlphaChars`, `formatByPattern` | no equivalent (vanity letters are converted when parsing; custom patterns are not supported) | | `getExampleNumber`, `getExampleNumberForType` | `PhoneNumber.GetExample(region, type)` (null when the region has none) | | `getExampleNumberForNonGeoEntity` | `PhoneNumberParser.Default.GetNonGeographicExample(callingCode)` | | `isAlphaNumber` | `PhoneNumberParser.IsVanityNumber(text)`; `PhoneFacts.Vanity` | | `getLengthOfGeographicalAreaCode`, `getLengthOfNationalDestinationCode` | `GetAreaCodeLength(number)`, `GetNationalDestinationCodeLength(number)` | | `isNumberGeographical`, `canBeInternationallyDialled` | `IsGeographical(number)`, `CanBeInternationallyDialled(number)` | | `getNddPrefixForRegion` | `GetNationalPrefix(region)` | | `isNANPACountry`, `isMobileNumberPortableRegion` | `IsNorthAmericanPlan(region)`, `IsMobileNumberPortableRegion(region)` | | `isNumberMatch` and `MatchType` | no equivalent: compare `number == other` (canonical form) or `number.Key` (E.164 plus extension) | | `findNumbers(text, region)` and `PhoneNumberMatcher` | `PhoneNumber.FindAll(text, region, leniency)`; the engine form `PhoneNumberMatcher.Default.FindAll(...)` / `TryFindNext(...)` on spans; matches carry `Start`, `End`, `RawString`, `Number` | | `Leniency.POSSIBLE / VALID / STRICT_GROUPING / EXACT_GROUPING` | `PhoneMatchLeniency.Possible / Valid / StrictGrouping / ExactGrouping` | | `AsYouTypeFormatter` (`getAsYouTypeFormatter(region)`, `inputDigit`, `inputDigitAndRememberPosition`, `getRememberedPosition`, `clear`) | `new AsYouTypeFormatter(region)`, `InputDigit(char)`, `InputDigitAndRememberPosition(char)`, `RememberedPosition`, `Clear()` | | `ShortNumberInfo` `isPossibleShortNumber`, `isValidShortNumber` | `ShortNumberInfo.Default.IsPossible(number)`, `.IsValid(number)` and the `...ForRegion(number, region)` forms | | `getExpectedCost`, `ShortNumberInfo.ShortNumberCost` | `GetExpectedCost(number)`, `GetExpectedCostForRegion(number, region)`; `ShortNumberCost` | | `isEmergencyNumber`, `connectsToEmergencyNumber` | `IsEmergencyNumber(text, region)`, `ConnectsToEmergencyNumber(text, region)` | | `isCarrierSpecific`, `isSmsServiceForRegion`, `getExampleShortNumber` | `IsCarrierSpecific`, `IsSmsServiceForRegion`, `GetExample(region)` | | `PhoneNumberOfflineGeocoder.getDescriptionForNumber(number, locale)` | `PhoneGeocoder.Default.GetDescription(number, "en")` (a BCP 47 tag, package `RecursiveAI.OneContact.Phones.Geo`) | | `PhoneNumberToCarrierMapper.getNameForNumber`, `getSafeDisplayName` | `PhoneCarrierMapper.Default.GetName(number, tag)`, `.GetSafeDisplayName(number, tag)`; the carrier is the one a range was first assigned to, so read the caveat in [phones-geo.md](https://recursiveai.org/onecontact/docs/phones-geo/) | | `PhoneNumberToTimeZonesMapper.getTimeZonesForNumber` | `PhoneTimeZoneMapper.Default.GetTimeZones(number)` | | Loading other metadata (`MetadataLoader`, a replaced metadata file) | `new PhoneNumberParser(PhoneMetadata.LoadXml(stream))`, which reads the same XML format with size limits; see [phones.md](https://recursiveai.org/onecontact/docs/phones/) | | A `PhoneNumber` as a dictionary key or duplicate check | `number.Key` (E.164 plus extension) or `Contacts.SameKey` | Concepts that differ: - **Metadata** is compiled into the package from libphonenumber's published metadata files at a pinned release; [phones.md](https://recursiveai.org/onecontact/docs/phones/) says which. A newer release arrives as a patch release, or load it yourself with `PhoneMetadata.LoadXml`. - **Offsets.** `number.Source` is the span of the number inside the text you parsed; matcher results carry `Start` and `End`. - **Batch.** `PhoneNumberParser.ParseMany(inputs, region)` parses a sequence without exceptions. ## libpostal → OneContact libpostal gives two calls: `parse_address` (free text to labelled components) and `expand_address` (free text to normalized variants), plus near-duplicate hashing. OneContact folds them into `PostalAddress` (parse, standardize, render, validate, key) and `AddressExpander`. The big differences: OneContact reads rules first; in 1.0 that is all it does, and a text the rules cannot read is refused with a reason (small model packages for what the rules leave open follow in 1.1), it has no multi-gigabyte data directory, and it renders addresses back in each national postal authority's own style. | libpostal | OneContact | |---|---| | `parse_address(text)` with optional `language` and `country` options | `PostalAddress.Parse(text)`; with a country, `PostalAddress.TryParse(text, Region, out value, out detail)`. Without a country OneContact infers it from the evidence in the text (`Flags` has `CountryInferred`) or reports `CountryUndetermined`; it never guesses | | Components come back as `(label, value)` pairs | Properties on `PostalAddress`; offsets and confidences through `ParseDetail.Fields` and `PostalAddress.RuleDetail` | | `house` (a venue or building name) | `Organization` / `Recipient` lines | | `house_number` | `StreetDetail` part `StreetPartRole.HouseNumber` | | `road` | `StreetDetail` parts `PreDirectional`, `StreetName`, `StreetType`, `PostDirectional`; the whole line is in `StreetLines` | | `unit` | parts `UnitDesignator` and `UnitNumber` | | `po_box` | street line of kind `PostOfficeBox` with parts `BoxDesignator`, `BoxNumber` (`AddressFacts.PostOfficeBox`) | | `level`, `staircase`, `entrance` | not separate fields; the text stays in the street line as written | | `suburb`, `city_district` | `DependentLocality` | | `city` | `Locality` | | `state_district`, `state`, `country_region` | `AdminArea` (the authority's state, province or territory code is the standard form) | | `postcode` | `PostalCode` | | `country` | `Country` (a `Region`) | | `category`, `near`, `island`, `world_region` | not fields | | `expand_address(text, languages=[...])` | `AddressExpander.Expand(text, ExpansionLanguages.English \| ExpansionLanguages.German, maxVariants)`; the languages are flags (`English`, `French`, `German`, `Dutch`, `Italian`, `Luxembourgish`, `Spanish`, `Portuguese`, `Catalan`, `Basque`, `Galician`, `Maltese`, `Irish`; English and French by default) | | Expansion options (which transformations to apply) | no per-transformation switches: the variants come back with the fully expanded one first, capped by `maxVariants` (default 16) | | Near-duplicate hashing and duplicate classification | `PostalAddress.Key` / `GetKey(KeyVersion)`: one deterministic string built from the parsed parts and the expansions; equal keys mean the same address under the documented rules (see [address-keys.md](https://recursiveai.org/onecontact/docs/address-keys/)). There is no probabilistic classifier | | `libpostal_setup()` and the data directory (about 2 GB) | nothing to set up; rules and dictionaries are compiled in (the core package is 1.4 MB); the model packages of 1.1 will be 10 to 23 MB | | Language classification (`libpostal_place_languages`) | not a call: the reader chooses languages from the country and the evidence | | Normalizing for display or labels | `address.ToString(AddressStyle.Postal / Display, AddressLayout.Label / SingleLine, includeCountry)`; `GetLabelLines()` and `ToSingleLine()` for the country's own field layout | | Checking an address is plausible | `address.Validate()` returns `AddressIssue`s from structural rules only (required fields, postcode pattern, postcode and region agreement). There are no reference-data lookups | Not equivalent: libpostal's training data and models are not used or read. The OneContact models (from 1.1) are trained on open registers, our own renderers and, for the default `Models` package, OpenStreetMap-derived addresses, with their provenance in the model packages. ## python-nameparser and probablepeople → OneContact python-nameparser (`HumanName`) splits a person name by rules, with customizable lists. probablepeople tags a person or corporation string with a trained CRF. OneContact does both jobs in `PersonName`, `CompanyName` and `NameParsing`: rules first, with the decision source reported (a small model for the unclear remainder follows in 1.1). | python-nameparser | OneContact | |---|---| | `HumanName(full_name)` | `PersonName.Parse(text)`; add `NameHints.ForCountry(Region)` or `NameHints.ForLanguage(tag)` for order and particles | | `name.title` | `Prefix` | | `name.first` | `Given` | | `name.middle` | `Middle` | | `name.last` | `Family`, with `Particle` kept separate (`van der`) and `SecondFamily` for Iberian names | | `name.suffix` | `Suffix` (generational) and `ProfessionalSuffix` (credentials) | | `name.nickname` | `Nickname` | | `name.full_name`, `str(name)`, `string_format` | `Canonical`; compose your own text from the parts | | `name.capitalize()` | done on parse: all-caps or all-lower input is restored (`NameFacts.CasingRestored`), traceable by offsets | | `name.original`, `name.unparsable` | `Source` span; `TryParse` returns `false` or an `OperationStatus` with a reason | | `name.as_dict()`, `initials()` | read the parts; no initials helper | | "Last, First" input | read as given order `FamilyCommaGiven` (`PersonName.Order`) | | Customizing `CONSTANTS` (titles, prefixes, suffixes, conjunctions) | no runtime mutation of the lists. Titles, suffixes, particles and legal forms are compiled data; use `NameHints` for country and language | | `"John and Jane Smith"` (kept as one string) | `NameParsing.Split(text)` returns separate `PersonName`s that share the family name | | Equality of names | `==` compares canonical forms; `person.Key` is the aggressive key (folded diacritics, nickname equivalence, titles and suffixes dropped, see [name-keys.md](https://recursiveai.org/onecontact/docs/name-keys/)) | | probablepeople | OneContact | |---|---| | `tag(text)` returning `Person`, `Corporation` or `Household` | `NameParsing.Classify(text, hints)` / `NameParsing.Split(text, hints).Class`: `NameClass.Person`, `Company`, `Multiple` (a household of several people), `Unknown`. Unknown is an honest answer | | `parse(text)` returning `(token, label)` pairs | the parts of `PersonName` and `CompanyName`, each with a span | | `GivenName`, `MiddleName`, `Surname` | `Given`, `Middle`, `Family` | | `FirstInitial`, `MiddleInitial`, `LastInitial` | no separate label: an initial is a `Given`, `Middle` or `Family` part whose text is one letter | | `PrefixMarital`, `PrefixOther` | `Prefix` | | `SuffixGenerational`, `SuffixOther` | `Suffix`, `ProfessionalSuffix` | | `Nickname` | `Nickname` | | `And` | a split into several people (`NameClass.Multiple`) | | `CorporationName`, `CorporationNameOrganization` | `CompanyName.Base` | | `CorporationLegalType` | `CompanyName.LegalForm`: an ISO 20275 entity legal form with its ELF code (`ElfCode`) and canonical abbreviation; `LegalFormBeforeName`, `LegalFormAmbiguous` and `LegalFormInactive` flags | | Other corporation labels (branch, committee, ...) | not parts. DBA and c/o are split out as `TradingAs` and `CareOf`; a leading "The" is `LeadingArticle` | | Model confidence | `ParseDetail` per field: `Confidence` and `Source` (`Rule` or `Model`) | | Training your own model | from 1.1: the recipes in `recipes/` use the `Train` API of OneNLP; see [models.md](https://recursiveai.org/onecontact/docs/models/) | Not equivalent: neither library's data or models are used. OneContact never infers gender, origin or a salutation from a name. # Parsing, formatting and JSON: what the types promise **Parse overloads.** `PhoneNumber`, `PostalAddress`, `PersonName` and `CompanyName` all take their context as a second argument: `Parse(text, Region)` and `TryParse(text, Region, out value)` on strings and spans. For names the region is shorthand for `NameHints.ForCountry(region)`; `NameHints` also takes a language. `Region.Unknown` means no context. **A successful result is usable.** `PostalAddress.Parse`/`TryParse` fail with `address.country-undetermined` when the country is neither given nor told by the content (a postcode form several countries share, no region name). The status-returning `PostalAddress.TryParse(span, out value, out detail)` is the low-level engine and keeps the partial reading: `Done`, the fields, `AddressFacts.CountryUndetermined`, `Country` unknown. `Contacts.Normalize` follows the happy path. **Reasons.** `ReasonCode.Id` (`phone.too-short`) is the stable text for logs and wire; `ReasonCode.Value` is the same code as the `ReasonValue` enumeration for `switch`/`case`. The enumeration value is the kind times 65536 plus the number. Keep a default branch: later versions add members. **Regions.** `Region` holds assigned ISO 3166-1 alpha-2 codes plus `AC`, `TA` and `XK`. `Region.Parse("ZZ")` is `Region.Unknown` and prints as `ZZ`; `UK` and other unassigned codes are refused (`invalid-region`). Whether a data set has an entry for an assigned region stays that data set's answer (`phone.unknown-region`, `Coverage`). **Formats.** `PhoneNumber` implements `IFormattable`: `C` (default) is the canonical form, `E` E.164 without extension, `I` international, `N` national, `R` RFC 3966, in either letter case, so `$"{number:N}"` works. Any other specifier throws `FormatException`. The default value formats as `""`. The other value types define only `C`. **Explicit metadata and models.** A `PhoneNumber` remembers the `PhoneNumberParser` that parsed it when that parser's metadata is not the built-in one. Models come from the registry (a referenced model package registers itself) unless you pass one: `ContactModels.Use(address, name, spans)` for a scope (`Use(null, null)` reads by rules only), or the `model` argument of `PostalAddress.TryParse(span, country, model, ...)`, `RuleAddressParser.TryParse(..., model, ...)`, `NameParsing.Classify`/`Split(..., hints, model)`. A `null` model means no model, not the registered one. **JSON.** Each value type is a JSON string holding its canonical form, except where noted. - `PhoneNumber`: canonical text. The attribute converter reads international numbers only; add `new PhoneNumberJsonConverter(Region.DE)` to `JsonSerializerOptions.Converters` to read national numbers too. - `EmailAddress`: the canonical form, or `"Display Name" ` when the value has a display name, so the display name round-trips. - `WebAddress`: the canonical form, which leaves out the fragment and tracking parameters; they do not come back. `CompanyName`, `PersonName`: canonical text; a legal form chosen with a hint is found again by parsing with the same hint. - `PostalAddress`: one object shape (`canonical`, `country`, `language`, the fields, `streetLines`, `flags`, `sourceStart`, `sourceLength`; parsed addresses add `text` and `hint`). Names are fixed camelCase. A plain JSON string is also read, as free text; add `new PostalAddressJsonConverter(Region.ES)` to `JsonSerializerOptions.Converters` to read strings as text of a country (an object carries its own stored hint). - `Region`: the two-letter code; `ZZ` is `Region.Unknown`. Errors are `JsonException`s whose message names the reason id and the byte offset of the token, with the `ContactParseException` as `InnerException`; the serializer adds the path and line. **Fragments.** `WebAddress` has no `Fragment` member. The canonical form, the key and `==` leave the fragment out, and a value does not keep it, so a property would be empty for every parsed value or would make a value carry text its canonical form hides. Keep the original text when you need a link that scrolls to a section; `WebFacts.FragmentRemoved` tells you there was one. **Extraction names.** `ContactExtractor` has two kinds of members, told apart by their names. The static `Extract(...)` overloads are the happy path: they take HTML, bytes or a set of pages, use default options and return the records. An instance (`new ContactExtractor { ... }`) carries options (`Sources`, `DefaultRegion`, `Limits`) and has `ExtractHtml`, `ExtractText`, `ExtractSite`, `ExtractSiteAsync` and `Extract(Document)`, which return the fuller `ExtractResult` with flat fields and provenance. The static and instance `Extract` overloads differ in their parameter types and are never ambiguous; the format-specific instance names exist so a string is never guessed to be HTML or text. **Limits.** Names and companies longer than `NameParsing.MaxInputLength` (1,000 UTF-16 code units) are refused (`person.too-long`, `company.too-long`). Extraction input limits are `ContactExtractor.Limits` (see extraction.md). # Phones Normative notes for the phone core. Public API: `PhoneNumber` (value type) and `PhoneNumberParser` (engine with explicit `PhoneMetadata`) in `RecursiveAI.OneContact.Phones`. ## Metadata ingest `tools/Ingest.LibPhoneNumber` converts the four pinned upstream files (`upstream/libphonenumber/PIN`: tag v9.0.43, commit 86ea33cee5c22ab7e23b3de368e7cc7fa439a2b7) into the compiled-data resource `OneContact.Data.libphonenumber.ocd` at build time. `tools/fetch/libphonenumber.sh` fetches the files by exact path and checks their SHA-256 against the PIN; the build fails on a missing or altered file. The same input gives byte-identical output. - Tables `main` (from `PhoneNumberMetadata.xml`), `short` (`ShortNumberMetadata.xml`) and `alt` (`PhoneNumberAlternateFormats.xml`) use one codec (`Phones/Metadata/PhoneMetadataCodec.cs`) over the XML's own structure. Whitespace inside pattern text is stripped, as the files' comments say it is not significant. - `phonemetadata.proto` is a format document only; nothing is generated from it. - Territory inheritance: a territory without its own number formats, or with `mainCountryForCode`, resolves exactly as the metadata comments describe; `$NP` and `$FG` in format rules are expanded once at load; international formats fall back to the national format when no `intlFormat` is given. - A loader for newer metadata: `PhoneMetadata.LoadXml(Stream)`. It reads the same XML format, ignores any DTD (no entity expansion), and enforces bounds: 32 million characters, 4096 territories, 64 K characters per pattern, element nesting 64, repeat count 1000. Violations and malformed input throw `InvalidDataException`; nothing else escapes. All patterns are compiled when the metadata loads. ## Pattern matching Metadata patterns (about 3,000, plus any the user supplies) run on our own backtracking matcher (`Phones/Metadata/PhoneRegex.cs`) over code points, with a step budget so a hostile pattern cannot run unbounded. It supports the subset the metadata uses and the case modes the parsing rules need (exact, ASCII-insensitive, Unicode-folding). `[GeneratedRegex]` cannot serve patterns that arrive at run time, and `new Regex` is not allowed in the library, so one engine serves both. `bench/OneContact.Bench` holds the comparison against source-generated patterns. Indicative measurement (BenchmarkDotNet short job, in process, AMD Ryzen 9 7950X3D, 4 cores reserved, machine load 16 to 31 during the run, .NET 10): matching 3 general-description patterns (US, GB, DE) over 5 inputs, 15 whole-input matches per operation, took 994 ns with our matcher and 235 ns with `[GeneratedRegex]`, no allocations in either. Generated patterns are about four times faster per match, so the matcher stays a tuning candidate (a first-character prefilter or a compiled automaton); it is kept because the metadata patterns are data, not source. End to end on the same run: `PhoneNumber.Parse` of a US national number 4.7 µs and 1.1 KB, parse plus international format 5.5 µs and 1.5 KB, parse plus validity of a GB number 5.9 µs and 1.3 KB. These are not publishable rows. ## Parsing Input classes accepted, each matching the reference's observable behaviour: surrounding junk before the first digit or `+`; punctuation; letters (vanity numbers: three or more letters map by the keypad, after upper-casing); full-width and other Unicode decimal digits; extensions (`ext`, `ext.`, `x`, `;ext=`, `#` and others); RFC 3966 `tel:` with `;ext=` and `phone-context` (global digits or a domain name; an empty context is refused); international prefixes of the default region; national prefixes and carrier-selection codes of the default region. - No region: only `+`-prefixed text parses; anything else fails with `phone.region-required`. A well-formed region with no metadata fails with `phone.unknown-region`. A `PhoneNumber` parsed by a `PhoneNumberParser` over your own metadata (or found by a matcher built on one) answers `Format`, `IsValid`, `Type`, `Region`, `GetPossibility` and the other members from that metadata; equality, the key and JSON use the canonical form only, and reading JSON always parses with the built-in metadata. - Text longer than 250 characters fails with `phone.too-long` before anything else. - `PhoneNumber.Source` is the extent of the number after leading junk is skipped (for `tel:`/`phone-context` input, the whole trimmed text). - String overloads keep the raw input (needed by original-format output and carrier codes); span overloads do not unless `keepRawInput` is set. - Reason codes: `phone.not-a-number`, `phone.invalid-country-code`, `phone.too-short-after-idd`, `phone.too-short`, `phone.too-long`, `phone.unknown-region`, `phone.region-required`; empty input uses the shared `empty` code. ## Canonical form, key, flags - Canonical: `+`, then `;ext=` when there is one (the RFC 3966 parameter form). - Key, `KeyVersion.V1`: `+` then `x` when there is one. The key never depends on the formatting of the input. - Flags (`PhoneFacts`): `Valid`, `HasExtension`, `Vanity`, `InternationalInput`, `NonGeographic`. ## Operations Validity, validity for a region, region of a number, number type, possibility (`PhonePossibility`: possible, local only, invalid country code, too short, invalid length, too long; also per number type), formats (E.164, international, national, RFC 3966, out-of-country from a calling region, for mobile dialing with and without formatting, original format, national with the carrier code), example numbers per region and type, supported types per region, geographical area-code and national-destination-code lengths, national prefix and international prefixes of a region. Batch form: `ParseMany`. ## Behaviours settled by experiment against the pinned reference The metadata and its comments leave these open; each was settled by feeding crafted input to the released reference jar and recording the output. All are covered by the parity run. - A region's preferred international prefix wins over the plain one when formatting out-of-country (Russia gives `8~10`). - An empty national prefix counts as none. - The supported types of a region never list "fixed line or mobile"; the metadata's shared pattern lists both. - Only ASCII digits and mapped letters pass through the keypad mapping used for vanity numbers; other scripts' digits count as digits elsewhere. Upper-casing precedes the mapping, so dotless i and long s map to the keypad 4 and 7; the Kelvin sign is not mapped. - The vanity test is ASCII-case-insensitive; the viability and extension tests fold Unicode case. - Local-only lengths of a general description are the union of all type lengths minus the national lengths. - A number whose national part is all zeros keeps and echoes its raw input when formatted; E.164 prepends `+` only when the code came from the default region. - A well-formed but unknown region and an unknown country code both report an invalid country code in the reference; we keep two reason codes and the harness maps both. - Possibility counts a local-only length as possible for the yes/no question. - The letters of an extension label never make a number a vanity number: for `tel:+12025550100;ext=204`, `+12025550100 ext. 204` and `1-800-FLOWERS ext 5` the reference's `isAlphaNumber` answers false, false and true (the last because of FLOWERS). The `Vanity` fact is decided after the extension is cut off. (`tools/Oracles/Phones/AlphaProbe.java`; the reference also answers false for a whole `tel:` string with letters, because that text is not a viable number for that call; the fact here looks at the number only.) ## As-you-type formatting `AsYouTypeFormatter` (namespace `RecursiveAI.OneContact.Phones`) follows one number being typed. `InputDigit(char)` returns the text to show after each key press; `InputDigitAndRememberPosition(char)` additionally marks the key for `RememberedPosition`, the index in the last output just after that digit; `Clear()` starts again. The formatter never validates and never throws on odd input. - Digits of any script count; a plus sign counts only as the first key. Any other key stops formatting: from then on the output is the input exactly as typed, until `Clear()`. - The first two keys echo. At the third the formatter tries the region's international prefix, then a national prefix, then picks the region's formats by the leading digits of the number (three digits narrow the candidates, each further digit narrows them again with the format's later leading-digits patterns) and fills the first format's template, whose room is the longest number the format can match. When a typed number outgrows its template the next candidate format takes over. - A plus sign or an international prefix switches to the international formats of the region and then, once the calling code is read, of the region that owns it. Typing the national prefix first keeps it in front of the groups. A national prefix found late (Madagascar's seven-digit pattern) is taken out and formatting restarts after it. - The formatter works from the main region of a calling code, so Canada formats with the NANPA data. Settled by experiment against the pinned reference (the data files do not say): - In a region that uses a national prefix, a number typed **without** the prefix may use formats whose national-prefix rule is empty or is only the first group, and formats that make the prefix optional; a number typed **with** the prefix may use formats that make it optional or that carry a national-prefix or carrier-code rule. A number with an international prefix may use any. (United Arab Emirates "0800..." stays unformatted, "800..." is grouped; Bolivia groups "022..." through the carrier-code rule; the United States seven-digit format is skipped after a typed "1".) - Typing the NANPA trunk "1" keeps national formats; it does not switch to international ones. - Gabon has a national prefix pattern but no prefix of its own and behaves like a region with a prefix. - A full-width plus sign at the start is accepted as a key but never becomes a plus; it is carried as an unusable character. ## Matcher `PhoneNumberMatcher` (and the happy-path `PhoneNumber.FindAll(text, region, leniency)`) finds numbers in running text. A candidate is a run of digits with up to four punctuation characters between digit groups, optionally led by brackets or a plus sign and followed by an extension; candidates that look like slash-separated dates, time stamps (`2012-12-22 20:12` followed by `:30`), page ranges or unbalanced brackets are dropped. A candidate that fails is split at slashes, bracketed starts, dashes, dots and spaces and the pieces are tried. `maxTries` counts rejected candidates and pieces. Leniency, each level including the one before: | Level | Accepts | |---|---| | `Possible` | parses, and the length is plausible for the region | | `Valid` | valid; not preceded or followed by Latin letters or a currency sign or `%`; `x` only as an extension or carrier-code marker; the national prefix present where the region's format asks for it | | `StrictGrouping` | valid and not more than one slash inside the national number, and each group of digits of the region's national format (or of an alternate format in the data) stays together in order; separators between groups may be missing | | `ExactGrouping` | the groups are exactly the region's (or an alternate format's), from the end; the first may follow a national prefix | Alternate formats (`PhoneNumberAlternateFormats.xml`) are applied wherever their pattern matches inside the national number, not only when it covers it, which the reference does too (observed: Cambodia's `(2)(2)(2)(2-3)` alternate groups the first nine digits of a ten-digit number and leaves the tenth). The alternates are the library's own data even when a parser is built over user metadata. Matches carry `Start`, `End`, the raw text and the parsed number (with its raw input kept). The low-level `TryFindNext(span, region, leniency, ref index, ref triesLeft, out match)` continues a search without allocating an iterator. ## Short numbers `ShortNumberInfo` answers possible / valid / expected cost / emergency / carrier-specific / SMS-service questions from `ShortNumberMetadata.xml`; examples come straight from the data. `Default` uses the compiled data; `LoadXml(PhoneMetadata, Stream)` reads a newer file with the usual bounds. - A short number of a region is possible when its length is one of the lengths listed by any of the region's short-number descriptions, valid when it is possible and matches the short-code description. Questions "for a region" answer false when the number's calling code is not the region's. - Cost: premium rate, then standard rate, then toll free, otherwise toll free for an emergency number and unknown for the rest. Over all regions of a calling code the answer is the most expensive one (premium, unknown, standard, toll free). - `IsEmergencyNumber` matches the whole text; `ConnectsToEmergencyNumber` also accepts extra digits after an emergency number, except in Brazil, Chile and Nicaragua. A leading plus sign is never an emergency number; surrounding punctuation is ignored. - `IsCarrierSpecific(number)` asks the first region of the calling code whose short codes contain the number (settled by experiment: a region whose data lists the number as carrier-specific is not asked when an earlier region also has it as a short code). - The reference does not expose example short numbers or the list of regions with short-number data, so those two are tested against the data file, not the reference. ## Parity method `tools/Oracles/Phones` generates cases per region from the metadata's example numbers and patterns plus mutations (prefixes, punctuation, extensions, wrong lengths, other regions' numbers, `tel:` URIs, letters, Unicode digits, garbage), runs them through a small Java driver of ours that calls the jar's public API, runs the same cases through the C# engine, and diffs per operation and region. The jar and a user-local JDK are black boxes on the build machine; see `tools/Oracles/Phones/README.md`. The as-you-type, matcher and short-number cases come from the same generator (`generate-extra`, answered by `ours-extra`, with `ExtraDriver.java` on the jar side; `run-extra.sh` runs all steps). As-you-type: for every region the example number of each type and numbers sampled from each type's pattern, typed nationally, with the national prefix, with `+` and the calling code, with the region's international prefix, with punctuation typed in, as `+...`, plus random key presses. Each is checked after every key press, with a remembered position at the middle key, and typed again after `Clear()`. Matcher: numbers of every region in several formats and groupings embedded in prose templates, with dates, ids, digit runs, brackets and slashes, plus longer stitched texts, at every leniency and with three `maxTries` bounds. Short numbers: examples and pattern samples of every description of every region, mutated (one digit more or less, prefixed), asked in the region, in a sibling region of the calling code and in an unrelated region; emergency texts with trailing digits, spaces, plus signs and other scripts' digits. # Phones.Geo Normative notes for `RecursiveAI.OneContact.Phones.Geo`: the geocoder, the carrier mapper and the time-zone mapper for phone numbers. The package references the core library only and carries its own compiled tables; the core library stays free of this data. **Carrier caveat.** The carrier mapper names the carrier a number range was *first assigned to*. Where number portability exists a number can have moved to another carrier, and no offline table can know that. Use `GetSafeDisplayName` when a wrong carrier would mislead: it answers only for regions where portability does not apply. Never treat the carrier as the current one, and never as proof of who operates a line. ## Public API (`RecursiveAI.OneContact.Phones.Geo`) | Type | Members | |---|---| | `PhoneGeocoder` | `GetDescription(number, languageTag)`, `GetDescription(number, languageTag, userRegion)` (checks validity), `GetDescriptionForValidNumber(...)` (two overloads, no validity check) | | `PhoneCarrierMapper` | `GetName(number, languageTag)` (mobile, fixed-line-or-mobile and pager numbers only), `GetNameForValidNumber(number, languageTag)`, `GetSafeDisplayName(number, languageTag)` | | `PhoneTimeZoneMapper` | `GetTimeZones(number)`, `GetTimeZonesForGeographicalNumber(number)`, `UnknownTimeZone` (`Etc/Unknown`) | Each type has a `Default` instance on the core library's default metadata and a constructor taking a `PhoneNumberParser`. Language tags are BCP 47. Results never depend on the current culture. Nothing is fetched: every table is embedded and read on first use. ## Data and ingest `tools/Ingest.LibPhoneNumberGeo` converts four pinned data sets into compiled table files (`OneContact.Data..ocd`), one converter per pin source. `tools/fetch/*.sh` fetch the files by exact path and check each SHA-256 against its `PIN`; the build fails on a missing or altered file, and the same input gives byte-identical output. | Source (`upstream/…`) | Pin | Licence | Compiled size | |---|---|---|---| | `libphonenumber-geocoding` (`resources/geocoding/**`) | tag v9.0.43, commit 86ea33ce | Apache-2.0 | 2.6 MB | | `libphonenumber-carrier` (`resources/carrier/**`) | same | Apache-2.0 | 162 KB | | `libphonenumber-timezones` (`resources/timezones/map_data.txt`) | same | Apache-2.0 | 21 KB | | `cldr-territories` (CLDR JSON 43.0.0 territory names) | c3349f36 | Unicode License v3 | 0.9 MB | Notices are packed under `notices/` (Apache-2.0 licence and the compiled provenance; the Unicode licence for CLDR). Prefix maps (geocoding, carrier) are tables named `g//` and `c//`. A table holds the sorted keys (digit prefixes, at most 32 digits) in blocks of 64 entries, each key delta-coded against its predecessor, with an index of block offsets and a pooled description list. A lookup binary-searches the block heads and scans one block, so only the tables a lookup touches are opened. The reader checks every offset and count and throws `InvalidDataException` for a truncated or inconsistent table; `Validate()` walks a whole map and also checks key order. Country names for the "user in another country" geocoder variant come from CLDR 43 territory names for every locale CLDR has, delta-pruned against the parent locale. The reference runs on a JDK whose locale data is CLDR 43 (CLDR 42 differed, for example in the name of Türkiye), hence that pin. ## Behaviour Settled by black-box experiments against the released jars (below), and held on generated cases. - **Table language.** The requested language's own table answers by longest key that starts the national number. When it has no such key, or the entry is empty, the English table answers the same way. Chinese, Japanese and Korean never fall back to English. Chinese with region TW, HK or MO uses the traditional-script table when the calling code has one, otherwise the simplified one; other script subtags are ignored. Tags are BCP 47 only (`zh_TW` is not a tag, so it answers in English). The legacy codes `iw`, `in`, `ji` mean `he`, `id`, `yi`. A language with no tables (for example `mo`, `tl` for areas; any unknown tag) answers in English. - **Empty entries.** In the compiled non-English tables an entry that equals the English one is empty with its key kept, so a longest key that lands on it falls back to English instead of to a shorter key of the same language. Ingest reproduces this. - **Argentine mobile token.** Numbers of calling code 54 whose national number starts with 9 are looked up without that 9 in the geocoder and the carrier mapper. The time-zone map is unaffected. - **NANP.** Calling code 1 is looked up only when the national number has exactly ten digits. - **Geocoder.** A number of unknown type describes as empty. A place-bound number gets the prefix-lookup description, or, when nothing is found, the country name when the calling code or validity identifies exactly one region, else empty. A number that is not place-bound gets the country name by that rule. With a user region: the number's own region equal to the user's gives the full description; any other region gives that region's name in the requested language; an unknown region gives empty. - **Carrier.** `GetName` answers only for mobile, fixed-line-or-mobile and pager numbers. `GetSafeDisplayName` answers empty where `PhoneNumberParser.IsMobileNumberPortableRegion` is true. - **Time zones.** An unknown number type gives `Etc/Unknown`. A place-bound number takes the zones of the longest key that starts calling code plus national number; any other number takes the exact calling-code key; a missing key gives `Etc/Unknown`. - **Country names.** The root names are the English names of the 249 ISO 3166-1 codes; `en` additionally has the extra codes (`AC`, `XK`, …). Locales the reference runtime lacks (35 of them, for example `tn`, `ss`) answer in English. One right-to-left override character in the Adlam name of Botswana is removed. ## Experiments Each file under `tools/Oracles/PhonesGeo/probes/` is a crafted input list run through the jar driver (`GeoDriver.java`) and compared with our output; the conclusions are the behaviours above. | File | Question | Conclusion | |---|---|---| | `e1-basics` | descriptions across languages and user regions | prefix lookup per language; foreign user gets the country name | | `e2-fallback` | what a language without an entry returns | English table, except zh/ja/ko | | `e3-zh` | Chinese variants for CN, TW, HK | region picks the traditional table | | `e4-types` | description by number type | place-bound versus country-name rule | | `e5-token`, `e6-tokens` | mobile tokens (AR, MX) | only AR 9 is skipped | | `e7-locales` | which locales the runtime knows | CLDR 43 country names; 35 missing locales answer in English | | `e8-valid` | calling codes shared by several regions | unique-valid-region rule | | `e9-lengths` | NANP number lengths | ten digits only | | `e10-ko-keys` | empty entries in non-English tables | key kept, text empty, falls back to English | ## Parity run `tools/Oracles/PhonesGeo/run.sh [seed] [prefixes per calling code]` generates cases, runs the jars (libphonenumber 9.0.43, geocoder 3.43, carrier 2.43, prefixmapper 3.43, Temurin 21.0.13) and our mappers, and compares nine fields per case: description with and without a user region, for-valid-number forms with and without a user region, carrier name, for-valid-number carrier name, safe carrier name, time zones, and geographical time zones. The cases are example numbers of every region in every ISO language and some odd tags, plus numbers from the data's own prefixes in every calling code, plus mobile and toll-free numbers per region in all 612 CLDR locales. # Email addresses `EmailAddress` reads one mailbox: an `addr-spec` (`jane@example.com`) or a `name-addr` (`Jane Doe `), and gives a canonical form, a versioned key, flags and offsets into the input. `EmailParser` is the low-level form (span in, `OperationStatus` and `ParseDetail` out, plus a batch form). ## What is read | Input | Rule | |---|---| | addr-spec, name-addr, angle-addr | RFC 5322 section 3.4 | | comments and folding white space | section 3.2.2; comments are dropped and reported by the `HasComment` flag | | quoted-string local parts, quoted-pairs | section 3.2.4; kept as written in the canonical form | | display names | section 3.2.5 phrase; quoted strings unquoted; RFC 2047 encoded words that are a whole atom decoded (UTF-8, US-ASCII, ISO-8859-1, ISO-8859-15, windows-1252; others stay as written); adjacent encoded words join without the space between (RFC 2047 section 6.2) | | obsolete forms | section 4 and 4.4: read and flagged `ObsoleteSyntax`, never refused. They are white space or comments around the dots of a local part or domain, a quoted word mixed with unquoted words in a local part, a route (`<@a,@b:user@example.com>`, dropped), an unquoted `.` in a display name, obsolete control characters in quoted strings, comments and domain literals, several folds in a row | | UTF-8 | RFC 6532 section 3.2: every non-ASCII character is atext, qtext, ctext and dtext. An unpaired surrogate is refused (`email.invalid-character`) | | domains | RFC 5321 section 4.1.2 labels (`Let-dig [Ldh-str]`) after UTS 46 processing; domain literals of section 4.1.3 | Groups (`Name: a@x, b@y;`) and address lists are not one mailbox: they fail with `email.invalid-display-name` or `email.trailing-text`. Surrounding white space (line breaks included) is ignored, as for every value type. ## Canonical form `@`. The local part is exactly what was written, quotes and quoted-pairs included (`"a b"@example.com`), with comments, folding and white space around dots removed. The display name is not part of the canonical form (`DisplayName` has it). The domain is lowercase and in A-label form: `info@Bücher.example` gives `info@xn--bcher-kva.example` (`UnicodeDomain` shows the U-label form). Processing is UTS 46 non-transitional with `UseSTD3ASCIIRules`, `CheckHyphens`, `CheckBidi` and `CheckJoiners` on (IDNA2008 strictness, RFC 6531 section 3.3); the ß of `faß.de` is kept (`xn--fa-hia.de`). ASCII domains without `xn--` labels skip IDNA and are only lowercased and checked against the host-name rules. A domain is refused when a label is empty, starts or ends with a hyphen, holds a character a host name cannot hold, or when the last label is all digits (RFC 3696 section 2, which also refuses a bare IPv4 address). Domain literals: `[192.0.2.1]`, `[IPv6:2001:db8::1]` (hex lowercased, tag spelled `IPv6:`); a malformed IPv4 or IPv6 literal is refused (`email.invalid-address-literal`); other literal text is valid RFC 5322 and is carried with a warning (`email.not-smtp-literal`). The canonical form parses back to itself. ## Limits (flags, not refusals) `ExceedsLengthLimit` and a warning issue when the local part is over 64 octets (RFC 5321 section 4.5.3.1.1), the domain over 255, a label over 63 (RFC 1035 section 2.3.4), or the address over 254 octets (RFC 3696 erratum 1690). Octets are UTF-8 bytes of the canonical form. Input is limited only by memory; comment nesting is limited to 64 levels. ## Flags | Flag | Meaning | |---|---| | `RoleAccount` | the unquoted local part, up to a `+`, case-folded, is on the role list: RFC 2142 names (abuse, noc, security, postmaster, hostmaster, usenet, news, webmaster, www, uucp, ftp, info, marketing, sales, support) and common contact-page names (admin, administrator, accounts, accounting, billing, careers, contact, customerservice, customer-service, customer_service, enquiries, feedback, finance, hello, help, helpdesk, hr, inquiries, jobs, legal, mail, mailer-daemon, media, newsletter, no-reply, no_reply, noreply, nobody, office, orders, payroll, press, privacy, recruiting, registrar, root, sales, security, service, spam, team) | | `FreeProvider` | the domain is on our list of free public mailbox providers (several dozen domains: gmail, googlemail, yahoo and its country domains, ymail, rocketmail, outlook, hotmail, live, msn, aol, icloud, me, mac, proton, protonmail, pm.me, gmx, web.de, mail.com, zoho, yandex, mail.ru, fastmail, tuta, hey, qq, 163, 126, naver and others). ISP mailboxes are not on it | | `Disposable` | the domain, or a parent of it that has at least two labels, is on the `disposable-email-domains` list (CC0, pinned in `upstream/disposable-email-domains/PIN`, compiled by `tools/Ingest.DisposableEmail`). Domains are compared as A-labels | | `PossibleTypo` | the domain is a known mistyping of a big provider (`gmial.com`, `gmail.con`, `hotmial.com`, ...: the provider names gmail, hotmail, yahoo, outlook, icloud, live, aol, msn and protonmail with the top-level typos con, cm, om, vom, xom, cmo, ocm, cim, c0m, comm, coom, plus a list of name typos). `EmailAddress.TypoSuggestion` gives the address with the likely domain; nothing is ever changed | | `ObsoleteSyntax`, `HasComment`, `QuotedLocalPart`, `HasDisplayName`, `DomainLiteral`, `SingleLabelDomain` | facts about the text | | `InternationalizedLocalPart` | the local part has non-ASCII characters (delivery needs SMTPUTF8, RFC 6531) | | `InternationalizedDomain` | the domain went through IDNA (non-ASCII, or an `xn--` label) | ## Key, `KeyVersion.V1` `EmailAddress.Key` / `GetKey(KeyVersion.V1)` / `EmailKey.Compute`. Rules, in order: 1. **Unquote.** A local part written as one quoted string whose text (quoted-pairs resolved) is a valid dot-atom loses its quotes: `"john"@x` and `john@x` are one mailbox (RFC 5321 section 4.1.2). Otherwise the quoted form stays (and is case-folded below). 2. **Fold.** The local part is lowercased with the invariant culture and, if it has non-ASCII characters, composed to NFC (our own implementation of UAX #15 over Unicode data, so the result does not depend on the platform). 3. **Provider rules.** Only for these exact domains, and only when the local part is not left empty: | Domains | Rule | Provider page (read 2026-11-05) | |---|---|---| | `gmail.com`, `googlemail.com` | drop everything from the first `+`; drop all dots; the domain becomes `gmail.com` | "Dots don't matter in Gmail addresses", support.google.com/mail/answer/7436150; "Send emails from a different address or alias", section "Use Gmail aliases", support.google.com/mail/answer/22370; gmail.com and googlemail.com being one service: Gmail Help Community, "Confused email addresses", support.google.com/mail/thread/378507106 (a community answer; Google's help centre has no page on it) | | `outlook.com`, `hotmail.com`, `live.com` | drop everything from the first `+` | "Send email from a different address in Outlook.com", section "Using site-specific email addresses in Outlook.com", support.microsoft.com/en-us/outlook/send-email-from-a-different-address-in-outlook-com | | `yahoo.com` | drop everything from the first `-` (`basename-keyword`) | "Create, use, edit, or delete disposable email addresses in Yahoo Mail", help.yahoo.com/kb/SLN28338.html | | `fastmail.com` | drop everything from the first `+`; `anything@user.fastmail.com` (one subdomain level) becomes `user@fastmail.com` | "Plus addressing and subdomain addressing", www.fastmail.com/help/receive/addressing.html | 4. **Everything else** (unknown domains, domain literals): rules 1 and 2 only. The domain part of the key is the canonical domain. iCloud has no rule: Apple's help pages do not document `+` addresses, so only rules 1, 2 and 4 apply to `icloud.com`, `me.com` and `mac.com`. The key is `@`. Re-reading a key as an address gives the same key. A published key version's output never changes: a new rule is a new `KeyVersion`. ## Reason codes `email.missing-at` 1, `empty-local-part` 2, `empty-domain` 3, `invalid-local-part` 4, `invalid-domain` 5, `invalid-display-name` 6, `invalid-quoted-string` 7, `invalid-comment` 8, `unbalanced-angle` 9, `trailing-text` 10, `invalid-address-literal` 11, `domain-idna-failed` 12, `numeric-top-level-domain` 13, `invalid-character` 14. Issues that do not stop the parse: `local-part-too-long` 20, `domain-too-long` 21, `label-too-long` 22, `address-too-long` 23 (warnings); `obsolete-syntax` 24, `single-label-domain` 25, `possible-typo` 26 (info); `not-smtp-literal` 27 (warning). An empty input or empty `<>` is the shared `empty`. ## Offsets `EmailAddress.Source` is the mailbox without surrounding white space. `ParseDetail.Fields` has `display-name` (from the first to the last token, quotes included), `local-part` and `domain`, as ranges of the input, decided by rules with confidence 1. A failure's issue span points at the character that stopped the parse. ## Conformance `tests/OneContact.Tests/Email/corpus/` holds the cases (format in its README): every address example of RFC 5322 (Appendix A and the sections' grammar), RFC 3696 section 3 with its errata, RFC 2047 section 8, grammar-derived cases for RFC 6531/6532 (those RFCs print no example addresses), and a case for every provider rule and flag. The isemail list (BSD-3-Clause, pinned in `upstream/isemail/PIN`) is run through the parser with expectations from its categories; the 35 cases where our reading of the RFCs differs are listed with their clause in `isemail-overrides.cases`. # Web addresses `WebAddress` parses a website address with the WHATWG URL Standard and reports a canonical form, a comparison key, facts and offsets. Everything here is written from the URL Standard (living standard, last updated 19 October 2026), UTS #46, RFC 3492 and the Public Suffix List's published algorithm. Nothing depends on culture or the network. ## Parsing - The URL parser is the Standard's basic URL parser, state by state (`Web/UrlParser.cs`), with its host parser (domains through UTS #46, IPv4 and IPv6, opaque hosts), percent-encode sets and serializer. The Standard's setters are implemented as far as its conformance data needs them, internally. - Conformance is measured on the web-platform-tests data pinned in `upstream/wpt-url/PIN`: `urltestdata.json` (896 entries), `setters_tests.json` (278), `toascii.json` (87) and `IdnaTestV2.json` (2676). All pass. `tests/OneContact.Tests/Web/WptUrlTests.cs` runs them; the data and its BSD-3-Clause licence are under `tests/OneContact.Tests/testdata/wpt`. - Input text is trimmed of white space. A missing scheme is assumed to be `https` and the result carries `WebFacts.SchemeAssumed`. A scheme counts as written when the text starts with `scheme:` and the colon is followed by `//`, or the scheme is one that is written without slashes (`mailto`, `tel`, `sms`, `urn`, `geo`, `callto`, `skype`, `magnet`, `news`, `data`, `javascript`, `about`, `blob`) or a special scheme. So `localhost:3000/x` and `example.com:8080` are hosts with a port, not schemes. - A failed parse reports one of `web.host-missing`, `web.invalid-host`, `web.invalid-port`, `web.invalid-url`. - Offsets: `ParseDetail.Fields` holds `scheme` (only when written), `host`, `port`, `path`, `query` and `fragment` spans, in UTF-16 units of the input as given (tabs and newlines the Standard removes are skipped, so spans cover them). - Hardening limits that are not in the Standard: a host that is not ASCII and is longer than 4096 UTF-16 units fails (domain names are at most 253 octets; the limit keeps quadratic Punycode and normalization steps from being abused). ASCII hosts are never run through UTS #46, as the Standard says. ## Canonical form The serialization of the URL with, in this order: 1. host lowercased and processed by UTS #46 (non-transitional, `CheckBidi` and `CheckJoiners` on, `UseSTD3ASCIIRules` off), so internationalized hosts appear in their `xn--` form; IPv4 numbers normalized to dotted decimal; IPv6 compressed; 2. default port left out (the Standard's rule); 3. tracking parameters removed from the query (below); a query that becomes empty disappears with its `?`; a query that was empty and had nothing removed stays; 4. the fragment removed. Credentials, path case and every other query parameter stay as written. Removing a fragment or tracking parameters is reported in `Flags`, never silent. ### Tracking parameters Names are matched ignoring ASCII case, on the name as written (not percent-decoded). Each name is on the list because a platform documents it; read 2026-11-05. | Names | Platform and page | |---|---| | `utm_source`, `utm_medium`, `utm_campaign`, `utm_term`, `utm_content`, `utm_id`, `utm_source_platform`, `utm_creative_format`, `utm_marketing_tactic` | Google Analytics 4, "Collect campaign data with custom URLs", support.google.com/analytics/answer/10917952 | | `gclid` | Google Ads, "About auto-tagging", support.google.com/google-ads/answer/3095550 | | `gbraid`, `wbraid` | Google Ads API, "Upload click conversions", developers.google.com/google-ads/api/docs/conversions/upload-clicks | | `msclkid` | Microsoft Advertising help, "Auto-tagging of Microsoft Click ID (MSCLKID)" | | `fbclid` | Meta, "ClickID and the fbp and fbc Parameters", developers.facebook.com/documentation/ads-commerce/conversions-api/parameters/fbp-and-fbc | | `ttclid` | TikTok Ads Manager, "About TikTok Click ID", ads.tiktok.com/resources/help/article/tiktok-click-id | | `twclid` | X Ads API, "Web conversions", docs.x.com/x-ads-api/measurement/web-conversions | | `li_fat_id` | LinkedIn Marketing API, "Enabling first-party cookies", learn.microsoft.com/en-us/linkedin/marketing/conversions/enabling-first-party-cookies | | `mc_cid`, `mc_eid` | Mailchimp developer documentation, "E-commerce", mailchimp.com/developer/marketing/docs/e-commerce/ | ## Key, version 1 `KeyVersion.V1`: `WebKey.Compute` over the canonical parts. A published version's output never changes; the golden fixture is in `WebAddressTests.KeyV1GoldenFixture`. 1. The scheme is left out for `http` and `https`; every other scheme is kept as `scheme:` (and `//` when the address has a host). 2. A domain host becomes its registrable domain (the public suffix plus one label, from the Public Suffix List with its private section), so `www.`, `m.` and any other subdomain do not tell two addresses of one organization apart. A root dot is dropped. A host with no registrable domain (an IP address, an opaque host, a name that is itself a public suffix) is kept whole. 3. A port that is not the scheme's default is kept as `:port`. 4. The path is normalized: ASCII letters lowercased; `%XX` escapes of unreserved characters (letters, digits, `-` `.` `_` `~`) decoded and the hex digits of the other escapes uppercased; runs of `/` collapsed; a final segment that is a default document (`index.html`, `index.htm`, `index.php`, `index.asp`, `index.aspx`, `default.htm`, `default.html`, `default.asp`, `default.aspx`) dropped; the trailing `/` dropped (so `/` alone is empty). 5. The query's parameters (already without tracking parameters) are sorted ordinally and joined with `&`; empty parameters are dropped. 6. Credentials and the fragment are not part of the key. 7. An address with an opaque path (`mailto:`, `tel:`) has no host to reduce: the key is the scheme, `:`, and the opaque path lowercased. ## Facts `WebFacts` reports what was found; it never changes the value: `SchemeAssumed`, `IpHost`, `InternationalizedDomain` (some label is `xn--`), `NonWebScheme` (not http or https; `IsWebsite` is false for those values: parsing accepts any WHATWG URL, including `javascript:` and `data:`, so check `IsWebsite` before using a value as a link), `HasCredentials`, `TrackingParametersRemoved`, `FragmentRemoved`, and the social-profile shapes. ### Social-profile shapes Rules we wrote from how each network shapes profile addresses (the length limits are the networks' usual handle limits, checked against nothing but the examples in our tests); a match says the address has the shape of a profile, not that it exists. Hosts match the domain and its subdomains, for `http` and `https` only. - LinkedIn (`linkedin.com`): `/in/`, `/company/`, `/school/`, `/showcase/`, `/pub/`. - X (`x.com`, `twitter.com`): a single segment of up to 15 letters, digits and underscores that is not a site page (`home`, `explore`, `search`, `settings`, `i`, `intent`, ...). - GitHub (`github.com`): a first segment of up to 39 letters, digits and hyphens that is not a site page, or `/orgs//...`. - Facebook (`facebook.com`, `fb.com`): a first segment that is not a site page (`sharer`, `login`, `events`, `watch`, ...), or `profile.php`, `people`, `groups`. - Instagram (`instagram.com`): a single segment of up to 30 letters, digits, periods and underscores that is not a site page (`p`, `reel`, `explore`, ...). - YouTube (`youtube.com`): `/@handle`, `/channel/`, `/c/`, `/user/`. - TikTok (`tiktok.com`): `/@handle` alone. - Mastodon-style: a single `@name` segment on any other host, except hosts where `/@name` means something else (`medium.com`, `substack.com`, `threads.net`, `threads.com`). ## Public Suffix List See `docs/psl.md`. `ContactKindDetector` treats text as a web address when it has `://`, starts with `www.`, or is a bare domain (optionally with port, path, query, fragment) whose last label is a top-level domain with a rule in the list's ICANN section, so `notes.txt`, `1.5` and `John.Smith` are not taken. # Public Suffix List `RecursiveAI.OneContact.Web.PublicSuffixList` answers two questions about a domain name: its public suffix and its registrable domain. ## Data - `public_suffix_list.dat` ships **unmodified**, byte for byte as published (pinned in `upstream/psl/PIN`: publicsuffix/list commit `aae37acca5d0f3d09e7c1fe4da9024c514674ac9`, SHA-256 `762b6cbd...3430`), packed under `notices/psl/` in the package with its MPL-2.0 licence text. Source Code Form is subject to MPL-2.0; the file is never edited, and our additions live in a separate file (`src/OneContact/Web/Data/psl-additions.txt`, same syntax, read after the list, counted as private-section rules; empty for now). - The ingest tool `tools/Ingest.Psl` compiles the same pinned file at build time into the table `rules`: one rule per line, sorted ordinally, as `[N|W|E][I|P]`: N plain rule, W wildcard rule (`*.rule`, stored without the star), E exception rule (`!rule`, stored without the bang); I and P say whether the rule is in the ICANN or the private section. Rules whose labels are not ASCII are stored in their `xn--` form (Punycode, RFC 3492), so lookups only compare ASCII. Each line of the list is read up to its first white space, as the list says. The same pinned inputs give the same bytes. - `tests/test_psl.txt` from the same commit (CC0) is run by `PublicSuffixListTests`: all 78 checks pass. ## Algorithm The list's published algorithm (publicsuffix.org/list, "Formal algorithm"): the labels of the domain are matched against every rule from the right; a wildcard label matches any one label; if several rules match, an exception rule prevails, otherwise the rule with the most labels; if none matches the rule is `*`; an exception's public suffix drops its leftmost label; the registrable domain is the public suffix plus the label in front of it. - Input may be Unicode or `xn--` and any letter case; labels are mapped with UTS #46 so `Bücher` and `xn--bcher-kva` find the same rule. The answer is lowercase and keeps the script the input used. - A name that is empty, starts or ends with a dot, or has an empty label has no answer (`null`). - `includePrivateSection: false` ignores the private section, so `user.github.io` has the public suffix `io`. - Lookups look at no more labels than the longest rule has, so a host with thousands of labels costs no more than a short one. # IDNA (UTS #46) data and processing Internationalized host names go through Unicode IDNA Compatibility Processing (UTS #46, non-transitional) the way the URL Standard asks for. Everything is in `src/OneContact/Text/Idna`. ## Inputs Pinned in `upstream/unicode-idna/PIN` (Unicode Character Database 18.0.0, Unicode License v3, `upstream/unicode-idna/LICENSE`): | File | Used for | |---|---| | `idna/IdnaMappingTable.txt` | status (valid, ignored, mapped, deviation, disallowed) and mapping of every code point | | `ucd/UnicodeData.txt` | canonical combining class, general category Mark, canonical decompositions | | `ucd/DerivedNormalizationProps.txt` | `Full_Composition_Exclusion` | | `ucd/extracted/DerivedBidiClass.txt` | Bidi_Class, for the rules of RFC 5893 | | `ucd/extracted/DerivedJoiningType.txt` | Joining_Type, for the ContextJ rules of RFC 5892 appendix A | 18.0.0 is the version the pinned web-platform-tests data (2026-11) is generated from: with 17.0.0 seven entries of `IdnaTestV2.json` fail on characters added in 18.0. The ingest tool `tools/Ingest.Idna` writes them as tables of the compiled-data file (`docs/architecture.md`); all numbers are little-endian: - `idna.ranges`: 12-byte entries `first u32, last u32, info u32`, sorted; `info = status | length << 3 | offset << 11`, where status is 0 valid, 1 ignored, 2 mapped, 3 deviation, 4 disallowed, and the mapping is `length` UTF-16 units at `offset` (in units) in `idna.map` (UTF-16 little-endian). Code points not listed are disallowed. - `ccc`, `mark`, `compexcl`, `bidi`, `joining`: 8-byte entries `first u32, (last | value << 24) u32`, sorted, non-overlapping, listing only non-default values (ccc not 0; mark 1; excluded 1; bidi and joining value below). - `bidi` value: 0 L, 1 R, 2 AL, 3 EN, 4 ES, 5 ET, 6 AN, 7 CS, 8 NSM, 9 BN, 10 B, 11 S, 12 WS, 13 ON, 14 LRE, 15 LRO, 16 RLE, 17 RLO, 18 PDF, 19 LRI, 20 RLI, 21 FSI, 22 PDI; code points the file does not list count as L. - `joining` value: 1 L, 2 R, 3 D, 4 C, 5 T; unlisted is U. - `decomp`: 12-byte entries `code point, first, second` of canonical decompositions of one or two code points (second is 0 for a singleton), sorted by code point. ## Processing `Uts46.TryToAscii` and `TryToUnicode` follow section 4 of UTS #46: map; normalize to NFC (our own implementation over the tables, UAX #15, so the result does not depend on the operating system's ICU); break at `.`; convert `xn--` labels with Punycode (RFC 3492) and validate every label (NFC, hyphen rules when asked, no leading mark, status valid or deviation, ASCII rules when STD3 is asked, ContextJ, the Bidi rule of RFC 5893 for Bidi domain names); then Punycode-encode non-ASCII labels. The URL host parser uses `CheckBidi` and `CheckJoiners` on, everything else off. Inputs longer than 4096 UTF-16 units fail: a hardening limit that is not part of UTS #46. # The structural address layer What the library knows about the shape of an address in every country the address data describes: which fields exist, in what order they are written, which are required, what a postal code looks like, and which subdivisions a country lists. It is data plus a small engine. It never looks anything up: a well-formed address that does not exist passes. ## Source The per-region address data of Google's libaddressinput, `testdata/countryinfo.txt`, at a pinned commit (see `upstream/libaddressinput/PIN`: repository, commit, SHA-256 of the file and of the licence). The file holds one `key={json}` line per node of a tree: ``` data the list of countries (property "countries") data/US a country data/US/CA a subdivision (administrative area) data/BR/AC/Acrelândia a locality; one level deeper is a dependent locality data/CA--fr the French variant of a country; subdivisions carry the same suffix (data/CA/AB--fr) data/ZZ the defaults: what a country that gives no value of its own gets examples/... sample addresses for a few countries (not compiled) ``` Licence: Apache-2.0 (the repository's LICENSE). The repository README also says its data is available under CC BY 4.0; both are credited in the packed notice. Fetch with `tools/fetch/libaddressinput.sh`. ### Properties All values are strings. Lists are separated by `~`, and the lists of one node are parallel (`sub_keys`, `sub_names`, `sub_lnames`, `sub_zips`, `sub_zipexs`, `sub_mores`, `sub_isoids`, `sub_lfnames`). | Property | Meaning | |---|---| | `key`, `name`, `lname`, `lfname` | The node's key, its name in the local script, its Latin name, its Latin full name. The country `name` is English in capitals. | | `lang`, `languages` | The main language, and the official languages (`~` separated). | | `fmt`, `lfmt` | The address format in the local script, and in Latin script. Codes: `%N` recipient, `%O` organization, `%A` street lines, `%D` dependent locality, `%C` locality, `%S` administrative area, `%Z` postal code, `%X` sorting code, `%n` line break. Anything else is literal text. | | `require` | Letters of the required fields (`ACSZ`). The country is always required. | | `upper` | Letters of the fields the postal service writes in capitals. | | `zip` | At country level a pattern every postal code matches. At subdivision level a pattern the start of the codes of that subdivision matches. | | `xzip` | At subdivision level a pattern the whole code must match, replacing the country's pattern. | | `zipex`, `sub_zipexs` | Example postal codes (`,` separates codes, `:` separates ranges). | | `sub_keys` | The keys of the subdivisions below the node. | | `state_name_type`, `zip_name_type`, `locality_name_type`, `sublocality_name_type` | What the country calls the administrative area (`state`, `province`, `prefecture`...), the postal code (`zip`, `postal`, `pin`), the locality and the dependent locality. Defaults: `province`, `postal`, `city`, `suburb`. | | `posturl`, `postprefix`, `isoid`, `sub_xzips`, `sub_xrequires`, `xrequire` | Carried through; not used by the engine. | ## Compiled tables `tools/Ingest.LibAddressInput` converts the pinned file into three tables of the compiled data file (container: `docs/architecture.md`). The same file always gives the same bytes. Version 1: - `props`: the property names that occur, in ordinal order, joined by a line feed. A property is referred to by its position. A reader ignores names it does not know. - `strings`: `u32 count`, `u32 offsets[count + 1]` (byte offsets into the data that follows, starting at 0), then the UTF-8 bytes. Every distinct value and node key appears once, in ordinal UTF-8 order. - `nodes`: `u32 count`, `u32 offsets[count]` (from the end of the offset array), then one record per node in ordinal order of its key: `varint key-string, varint property-count, (varint property, varint value-string)*`. Varints are 7 bits per byte, low first, at most 5 bytes. The reader checks every count and offset, that all strings are UTF-8, that every record parses and refers only to existing strings and properties, and that the keys are in order. Anything else raises `InvalidDataException`. ## Coverage `Coverage` is `Full` for the supported tiers and `Structural` for every other country in the data. Free text is parsed only for `Full` countries; for the others `PostalAddress.TryParse(text, country, ...)` refuses with the reason `address.free-text-refused`. The tiers: US with PR, VI, GU, AS, MP and UM; CA; MX, AU; GB with JE, GG and IM; AT, BE, HR, CZ, DK, EE, FI, FR, DE, HU, IE, IT, LV, LT, LU, MT, NL, PL, PT, RO, SK, SI, ES, SE; CH, NO, IS, LI; AD, MC, SM, VA; and the French overseas departments GF, GP, MQ, RE and YT. Greece, Bulgaria and Cyprus are never in a tier. ## Validation `PostalAddress.Validate()` returns at most one issue per field, in field order. The rules were settled from the data's documentation and by experiments with the reference implementation (below). 1. **White space.** A value is empty when it is made only of characters up to and including U+0020. Other white space (the no-break space, the ideographic space) is content. Values are trimmed of the same characters before they are compared. 2. **The format and the required fields** of a country are its own `fmt` and `require`, or the defaults' (`data/ZZ`) when it gives none. A country the data does not describe uses the defaults. 3. **The chain.** The country, the administrative area, the locality and the dependent locality are checked in that order, and checking stops at the first one that has a problem, because the levels below cannot be judged against a wrong level above. - Country: no country is `MissingRequiredField`; a country the data does not list (its `countries` list; `ZZ` is not a country) is `UnknownValue`. Either stops the chain. - A level with no value is `MissingRequiredField` when the country requires it, which stops the chain; otherwise the levels below have nothing to be looked up in. - A level with a value the country's format does not place is `UnexpectedField`, which stops the chain. - Otherwise, when the level above leads to a node that lists subdivisions, the value must be *known*: equal, ignoring case, to a key, a name or a Latin name in that node's lists; if not it is `UnknownValue` and the chain stops. When there is no list to check against any value is accepted. - A known value leads on to that subdivision's own node only when it equals a key exactly (case-sensitively), or equals a Latin name ignoring case. A value known by its local name only (`California` for `CA`, `강원` for `강원도`) is accepted but leads nowhere, so its own subdivisions and postal code prefix are not consulted. 4. **Sorting code, organization, recipient** are `UnexpectedField` when given and not placed by the format, and `MissingRequiredField` when empty and required. They do not depend on the chain. Street lines are `MissingRequiredField` when all are empty and `A` is required. 5. **Postal code**: not placed by the format and given: `UnexpectedField`; empty and required: `MissingRequiredField`; for a described country, not matching the whole-code pattern: `InvalidFormat`, where the whole-code pattern is the `xzip` of the most specific subdivision reached that has one, else the country's `zip`; and not matching the start-of-code pattern (`zip`) of the most specific subdivision reached that has one: `MismatchingValue`. A country the data does not describe has no pattern to check. ## Layout `GetLabelLines` and `ToSingleLine` (lines joined with `, `) lay an address out in the country's `fmt`, or its `lfmt` for the Latin script (the language tag names Latin, or the caller asks). A country with no `lfmt` uses its `fmt`. The rules: - Values are trimmed as above; a value empty after trimming is absent. - A field with no value leaves out the literal text written just before it in the format. - That literal text is also left out when nothing else has been written on the line yet, except for the first field of the line. - A line made of literal text only is kept; a line whose fields all have no value is dropped. - The street lines replace `%A` one per line (blank ones dropped). With more than one street line, whatever follows `%A` in the format starts a new line. Upper-casing (`upper`) is information only (`AddressRegionInfo.UpperCaseFields`); the structural layer never changes the case of a value. ## Postal code patterns The patterns are Java-style regular expressions in a small dialect: alternation `|`, groups `( )` and `(?: )`, classes `[...]` with ranges and `^`, `.`, the escapes `\d \D \w \W \s \S` and escaped punctuation, and the quantifiers `? * + {n} {n,} {n,m}` (greedy). Anchors, look-around, back-references, lazy and possessive quantifiers, named groups, `\p{..}` and nested classes are refused when a pattern is compiled. Matching is ignoring case, over a compiled program with a step budget and a recursion limit, so a hostile pattern cannot run long or overflow the stack. No `Regex` object is built from text at run time. ## Language variants A language tag given with the address selects the data variant `data/CC--tag` where the data has one (`CA--fr`, `CH--fr`, `CH--it`, `HK--en`); the tag is kept in lower case with `-` separators by the public API. The engine matches the tag exactly as it is given. ## Experiments with the reference The reference is the `common` module of libaddressinput built from the pinned commit (`tools/Oracles/Addresses/build-oracle.sh`) and driven through its public classes. Findings, each from crafted inputs (the reference was also given crafted variants of the data file): 1. The reference takes each country's `fmt`, `lfmt` and `require` from a table compiled into its jar (an older snapshot of the data), ignoring those entries in the data it is handed; it takes everything else (`zip`, `xzip`, `sub_keys`, `sub_names`, `sub_lnames`, the subdivision nodes) from the data. Crafting `fmt`/`require` of US, DE, FR and GB changed nothing; crafting `zip` and `sub_keys` of US and DE changed the verdicts. 2. Checking stops at the first problem among country, administrative area, locality and dependent locality (e.g. for a format without `%S`, a state and a dependent locality give only the state as unexpected). 3. A value is known by key, name or Latin name ignoring case; it leads on to a node by exact key or by Latin name ignoring case, never by local name (grid of 316 probes over the first four subdivisions of every country with subdivisions). 4. The postal code of a subdivision with `xzip` is checked against that pattern in place of the country's; the prefix `zip` used is the most specific one along the chain. 5. White space is the characters up to U+0020; a variant tag is looked up as written (`EN` finds no variant). 6. Layout: values trimmed; literal text dropped as above (11,152 field-subset probes over every format of its table matched the rules exactly); several street lines end the line. ## Where the library differs from the reference These are intended and listed for approval: 1. The shipped data is the pinned file, whose `fmt`, `lfmt` and `require` are newer than the reference's built-in table (the two differ in the format of 66 countries and the required fields of 19). The parity test gives the library the pinned data with those three entries replaced by the reference's. 2. The Latin layout of a country with no `lfmt`: the reference uses its default layout (street lines and locality only); the library uses the country's own `fmt`. 3. A language tag with a Latin script subtag (`zh-Hans-Latn`) selects the Latin layout; the reference recognises only two-part tags such as `ja-Latn`. 4. The public API lower-cases language tags and turns `_` into `-` before they select a data variant. ## Parity test `tools/Oracles/Addresses/make-reference.sh` regenerates `testdata/addresses-structural/`: the cases (from the pinned data and the reference's format table by `tools/Oracles/Addresses/CaseGen`, deterministic), the reference's answer to each, and a record of what produced them. `AddressStructuralParityTests` compares problems per field and label lines for every case. # Address rules for the United States and Canada How a free-text United States or Canadian address is read, standardized and rendered. The code is `RuleAddressParser` (reading), `StreetLineParser` (one delivery line), `AddressRenderer` (the postal and display styles) and `AddressExpander` / `AddressKey` (expansion and key). The rules are those of USPS Publication 28 (October 2024) and the Canada Post Addressing Guidelines (updated 2021-06-14, English and French editions); the rule tables are plain tab-separated files under `src/OneContact/Addresses/Data/US` and `.../CA`, so a new country is new data, not new code. ## Reading Input is one line (parts separated by commas, or by nothing) or several lines. The last line is found from the right: 1. A postal code: a ZIP or ZIP+4 (`22042`, `22042-1441`, nine digits get their hyphen), or a Canadian code (`A1A 1A1`, the letters D F I O Q U are never used and W and Z never start one). 2. A state, territory or province: a two-letter code or a name (English and French), up to four words. A two-letter code with no postal code beside it counts only when written in capitals. 3. The city: what is left of the line. When nothing is left the line above, if it is not a delivery line, is the city. Without commas the delivery line and the city run together (`123 MAIN ST APT 4 SPRINGFIELD`). The rules end the delivery line after the first street suffix (the last of consecutive suffix words, Publication 28 section 234.2), a one- or two-letter directional, and any unit pairs. Where no suffix is found, or the street type comes before the name (French), the rules cannot tell and the parse returns `address.model-required`: a trained model would be needed. With commas or line breaks no model is needed. **Country** comes from the content: a ZIP, a postal code, a state or province, an explicit country name (`USA`, `United States`, `Canada`). The optional hint is used only when the content is silent; a hint that disagrees with the content is reported as warning `address.country-conflict` and the content wins. A hint other than the United States or Canada is `address.unsupported-country`. With no content and no hint: `address.country-undetermined`. Confidence: 0.99 when postal code and state or province agree, 0.93 for a postal code alone, 0.85 for a state alone, 1 for a written country name, 0.9 for a hint. **Lines above the delivery line** (recipient, organization, attention, `MSC` lines) are kept as written (`LeadingLines`), with a postal-style form that is upper case with periods and commas removed (Publication 28 section 222). They are never parsed as names. **Nothing is invented.** No city from a postal code, no province from a forward sortation area, no `CALLE` or `APT` added, no unit designator guessed for a bare `#`. ## Delivery lines | Kind | Rule | |---|---| | Street | house number, optional fraction, pre-directional, name, suffix, post-directional, units. Parsed from the right (Pub 28 sections 231 to 237): a trailing directional (when a name remains), the suffix (when a name remains before it), a directional between name and suffix is part of the name and spelled out (233.3), two directionals in a row make one (`N E` is `NE`) except north-south and east-west pairs, a suffix word immediately before the suffix is spelled out (234.2), hyphens in street names become spaces (232), hyphens in house numbers stay (appendix D). `COUNTY ROAD`, `STATE HIGHWAY` and similar are names (237). | | Units | `APT`, `STE`, `UNIT`, `BLDG`, `FL`, `RM`, ... from Pub 28 appendix C2 (`#` followed by a space and the number, 213.2); Canada: `APT`, `SUITE`, `UNIT` (English), `APP`, `BUREAU`, `UNITÉ` (French); a Canadian unit before the civic number (`10-123`) is kept before it. A unit on its own line above the street line stays there (213.3); a unit on a line between the street line and the city joins the street line. | | Box | `PO BOX n`; `CALLER`, `FIRM CALLER`, `BIN`, `LOCKBOX`, `DRAWER`, `APARTADO` become `PO BOX` (283, 293); a leading hyphen of the ZIP+4 file becomes a zero (282); `PMB n` stays (285). Canada: `PO BOX` or `CP` with station information (`STN`, `SUCC`, `RPO`, `COP`). | | Rural route, highway contract | `RR n BOX m`, `HC n BOX m`; `RFD`, `RD`, `RURAL ROUTE` become `RR`, `STAR ROUTE` and `HIGHWAY CONTRACT ROUTE` become `HC`; leading zeros drop; `BOX` stands for `BZN` and `#`; extra words move to a line above (241 to 255). Canada: `RR n` with station information. | | General delivery | `GENERAL DELIVERY` spelled out (261); Canada `GD` (French `PR`) with station information. | | Military | `CPR`, `OPC`, `PSC`, `UPR` or `UNIT` with its number, `BOX` and a box number (238.1); last line `APO`, `FPO` or `DPO` with `AE`, `AP` or `AA`. | | Canada, additional address | `SITE n COMP m` above the rural route identifier (4.2). | | Puerto Rico | `URBANIZACION` and its spellings become `URB`, `CONDOMINIO` `COND`, `RESIDENCIAL` `RES`, `EDIFICIO` `EDIF` (292, 293); the number may come last or as bloque and number and is printed first (295, appendix I5); `NO` and `SO` after the street are directionals kept as written (294). | Without a house number only the line just above the city is read as a street line, and only if it holds a street suffix: the guides print number-less examples (`BAY DR W`), but a name line higher up (`CASEY BROOKS`) is not an address. Every part carries the rule that decided it (`pub28-233.22`, `canada-post-4.4-unit-number`, ...), its text as written, its standardized text and its offsets. ## Renderings `Postal`: upper case, the standard abbreviations, one space between elements, no periods; last line `CITY ST ZIP` with two spaces before the ZIP or postal code on a label (Pub 28 section 224 prefers two, Canada Post section 2 requires two) and one on a single line. Domestic addresses carry no country line (`includeCountry` adds `USA` or `CANADA`). `Display`: the same tokens in natural title case: `123 N Main St Apt 4` and `Springfield, VA 22162`. Directionals, state and province codes and box and route words (`PO`, `RR`, `HC`, `GD`, `STN`) stay in capitals; ordinals take a lower-case ending (`5th`); a number followed by a letter (`123A`) keeps its capital. A leading line the caller wrote in mixed case is left as written. `McDonald`-style capitals are not reconstructed. `PostalAddress.Parse` and `PostalAddress.TryParse(text, country, ...)` use these rules and fill the field model (country, state or province code, locality, postal code, delivery lines, recipient for lines above them); `StreetDetail` and `RuleDetail` give the street parts and offsets, `ToString(AddressStyle, AddressLayout)` the authority renderings, and `Validate()` the structural checks of the field model (ZIP and state, forward sortation area and province, required fields), which are not repeated here. An address made with `Create` is read again from its label lines for those members. The JSON form of an address with fields is an object holding the canonical form, the fields and the flags, and reads back equal. The canonical form of a `PostalAddress` parsed from text is the `Postal` rendering on one line with `, ` between lines; reading it back gives the same canonical form and key. ## Field names and what the rules read with certainty Field names: `street.house-number`, `street.fraction`, `street.pre-directional`, `street.name`, `street.type`, `street.post-directional`, `street.unit-designator`, `street.unit-number` (the `4` of `4-123 Main St` is a unit number with the hyphen outside it), `street.box-designator` and `street.box-number` (also after a military unit or a rural route: `PSC 3022 BOX 684`, `RR 4 BOX 932`), `street.route-*`, `street.station-*`, `street.military-*`, `street.general-delivery`, `locality`, `admin-area`, `postal-code`, `country`, and `leading-line` for lines kept as written. The words of one street name are one `street.name` field, and the final period of an abbreviation (`R.R.`, `U.S.A.`) is inside its field. Forms read only when nothing else is possible: - the street words that run on into the city with no comma (`72 Dewey St Richford, Vermont`): words left after the street's last word, when no other city is given; - a unit designator with its number (or a single letter) at the start of the city words belongs to the delivery line above (`..., Unit H Altura, MN`); - a French civic number followed by a comma and a street type (`4291, AV Christophe-Colomb`); - words in front of a house number on the line above the city (two or more, or one when another line stands above): kept as a leading line when they hold no digit, street word or directional; - the United States spelling of a street type in a Canadian address (`LN` is read as Lane and written as Canada Post writes it). Forms refused with `address.model-required`: a suffix word right after the street's suffix that also starts city names (`1300 Riverside Dr Mount Vernon WA`, `500 Main St Fort Worth TX`), because rules cannot tell the street's end from the city's start. ## Reason codes (kind address, numbers 110 to 116; 100 and 101 belong to the structural layer) `model-required`, `country-undetermined`, `unsupported-country`, `country-conflict` (warning), `no-delivery-line`, `no-last-line`, `unplaced-text` (warning). ## Where the two guides disagree Listed so the choice is visible; the tests follow the rule stated first. - Pub 28 section 213.2 asks for a space after `#` (`# 72`); the section 285 figures print `#234` and `#4545`. The code writes `# 234`. - Pub 28 section 233.3 says a directional between name and suffix is spelled out (`BAY W DRIVE` is `BAY WEST DR`) but the figure of alphabet indicators prints `BAY E DR`, because the local unit knows `E` is a letter. The code spells out every directional (`BAY EAST DR`); `BAY A DR`, `B` and `C` are unaffected. - Pub 28 section 295: the figure that moves the number first drops the word `CALLE` (`150 JULIO ENRIQUE VIZCARRONDO`), while the other figures of the same section keep it (`A17 CALLE 1`). The code keeps `CALLE`. - Pub 28 section 292 prints `AVE WILSON CHURCHILL 123` as a sample address and section 295 calls a street name followed by a number incorrect. The code follows 295. - Pub 28 section 295 prints `RR3 BOX 3415` as correct, against section 241 (`RR ## BOX ##`) and the figure above it (`RR03 BOX 9800` incorrect). The code follows 241. - Pub 28 appendix A32 spells `1010 CLEAR STREET` out for a Canadian address; Canada Post section 4.4 abbreviates the street type. The code abbreviates. - Canada Post lists Place as `PL` in the English edition and `PLACE` in the French edition; both are accepted, the English form is printed. - The Canada Post English edition omits Northwest Territories from its province table; `NT` is added. - Pub 28 figures show one space before the ZIP although section 224 prefers two; the tests ignore the width of a run of spaces. ## Examples that cannot be standardized by rules - Pub 28 section 223: `W STOCKBRIDGE` and `NEWBERRY SPGS` are 13-character city abbreviations from the City State file. - Pub 28 section 236: a corner address (`5TH AND HIGH`) is replaced by a physical address from reference data. - Pub 28 section 284: `# 4567` to `UNIT 4567` needs to know the address is a post office street address. - Pub 28 section 245 and 254 figures with a station line moved above a box line. # Address rules: France and its overseas departments, Monaco, Belgium, the Netherlands, Luxembourg, Germany, Austria, Switzerland, Liechtenstein How the rule parser (`Addresses/Tier2`) reads and standardizes these addresses. The rules are data (`Addresses/Data/T2/*.tsv`) plus one reader; adding a country of this kind means adding rows. The sources are the postal authorities' free public guides, listed at the end. Paid standards (NF Z 10-011 itself, DIN 5008) were not bought or read; where La Poste's guides cite a standard, only what the guide says is used. ## What the rules decide 1. **Lines.** Several lines, or one line with commas (or a lone `/`, `|`, `-` between words) in place of line breaks. A trailing country name is a field of its own. 2. **Last line.** The last postcode that has a locality after it ends the address: five digits (FR, MC, DE), four (BE, AT, CH, LI, LU), four digits and two capital letters (NL). A country letter before the postcode (`L-1234`, `CH-3000`, `FL-9490`, `D-26133`, `F-75008`, `B-1000`, `A-1010`, `MC-98000`) names the country. Words before the postcode on the same line are a line of their own. Switzerland and Liechtenstein: a canton abbreviation after the locality is the admin area (UPU Switzerland sheet). 3. **Country.** From the country name, the letter prefix, a hint, a postcode that fits one country (Dutch form, `97xxx` France, `980xx` Monaco), or language cues for the shared forms (CEDEX, BP, French street types for France; Postfach, Straße, Weg for Germany; a canton abbreviation for Switzerland). Otherwise `address.country-undetermined`; a hint the postcode cannot belong to is `address.country-conflict`. 4. **Delivery lines.** From the first line that is a box line or a street line to the line above the last. Lines above it are kept as written (recipient, organization, building, floor). A line that starts with a place-in-building word (`Appartement`, `Bâtiment`, `Gebouw`, `Verdieping`, `Chez`, …) is never the street line. Lines after the street line are further delivery lines (a box, a village, a lieu-dit). A bulk receiver has no delivery line: name lines without digits and the postcode with its locality are the whole address (UPU Luxembourg and Germany sheets); this carries a warning `address.no-delivery-line`. 5. **Everything else** is refused with `address.model-required`. Nothing is invented: no locality from a postcode, no street from a name. ## Postal and Display `Postal` is the authority's style, `Display` the same tokens in natural title case. Both are built from the same decisions. | Country | Postal | Source | |---|---|---| | FR, MC, GP, MQ, GF, RE, YT | Every line in capitals without accents or punctuation (hyphen and apostrophe become a space); six lines, 38 characters at most; a street type is written in full and abbreviated (`AV`, `BD`, `RTE`, `ZI`, …) only when the line is longer than 38; SAINT and SAINTE of the locality line and of the line above it become ST and STE; CEDEX and its digits stay; BP, CS, TSA lines kept. | La Poste, Guide pratique courrier 2026 pp. 18-21; SP 8855 volume 2 §3.1, §3.3; UPU France sheet | | BE | Lines as written; locality in capitals; one space between postcode and locality; a house number directly after the street name, `21C`, `21/3`, `4-8`; box number after the literal `bus`, `bte` or `box` (`b`, `Bt`, `#`, `-`, `/` are rewritten, the language of the street name decides: `bte` for French types, `bus` for Dutch endings); `n°`, `nr` dropped; Postbus numbers without spaces; type abbreviated only when the line is longer than 50 (`Av.`, `Bd`, `str.`, `ln`, `stwg`, …). | bpost Belgian template §3-4; bpost MassPost guide 2025 §1 pp. 11-13; UPU Belgium sheet | | NL | Lines as written; postcode `1234 AB`, two spaces, locality in capitals. | UPU Netherlands sheet; PostNL's page on addressing foreign mail says the same (read as search-result text; the PDF could not be downloaded) | | LU | Lines as written; postcode printed `L-1234`; locality in capitals. | UPU Luxembourg sheet | | DE | Lines as written; five-digit postcode without country letter; locality in capitals; `Postfach` numbers in the groups the sender wrote; `Postlagernd`, `Packstation 101` lines kept; apartment after `//`. | UPU Germany sheet | | AT, CH, LI | Lines as written; locality in capitals; box lines `Postfach`, `Case postale`, `Casella postale`. | UPU Austria and Switzerland sheets (Liechtenstein: Swiss Post serves it, the Switzerland sheet applies) | Display writes an all-capital word in title case (particles `de`, `du`, `des`, `van`, `von`, `am`, `im` in lower case, `L'` and `D'` elided), keeps a word that has any lower-case letter as the writer wrote it, keeps CEDEX, BP, CS, TSA in capitals, and never restores accents the input lacks. The country line (English, in capitals for Postal) is printed only when asked. ## Where the guides disagree or stop short - La Poste asks for capitals without accents "dès la ligne 1" as a recommendation; its own examples (UPU France sheet) show lines 1-3 in mixed case. `Postal` follows the recommendation for all lines; the tests compare line 1-3 of the examples in that form. - The UPU Belgium sheet and the bpost template print the municipality in mixed case (`1000 Bruxelles`); the bpost MassPost guide 2025 (p. 12) asks for municipalities in capitals. `Postal` follows the MassPost guide; `Display` keeps the mixed case. - The MassPost guide abbreviates `Air Marshall Conninghamlaan 432` to `Conninghamln` as an example while stating that abbreviations are for lines over 50 characters; the example line is 31 characters. The code follows the stated rule. - The UPU France sheet abbreviates IMPASSE as `MP` in its type table; La Poste's SP 8855 gives `IMP`. The code uses `IMP`. - The Swiss sheet prints `Solothurnstrasse` in the diagram and `Solothurnerstrasse` in the example; the example is used. - The UPU Germany sheet says to write P.O. box numbers in two-digit groups from the last number (`Postfach 3 41 41`); the code keeps the groups the sender wrote. - The returned-address form of La Poste's guide (p. 22, lines 4 and 6 on one line separated by three spaces) is not read. ## Not covered by rules (these go to the model) Addresses without a postcode; a name and a street on one line; a number glued to the street word (`5Route`); `Haus G`-style additions after the number; `R` for `rue`; Austrian and Swiss addresses with several local-part conventions beyond the sheets. ## Structural checks `PostalAddress.Validate` uses the structural layer's postcode patterns for each country. The reader adds the postcode ↔ region rules the sheets state: a postcode starting `97` is French (overseas departments are part of La Poste's network), `980` is Monaco. It never looks anything up. ## Keys and expansion Keys use the same segments as the United States and Canada (country, one segment per delivery line, locality, admin area, postcode) with words folded and the country's languages' dictionaries applied: `Hauptstr` is `hauptstrasse`, `av` is `avenue`, and the box literals `bus`, `bte`, `box` are one word. Existing key outputs are unchanged. `AddressExpander` takes `German`, `Dutch`, `Italian` and `Luxembourgish` besides `English` and `French`; German and Dutch names also expand an abbreviated ending (`Hauptstr` to `Hauptstrasse`) from libpostal's concatenated-suffix lists, keeping only abbreviations of three letters or more. ## Sources La Poste: Guide pratique courrier, January 2026 (laposte.fr); Documentation technique système client presse, Adressage SP 8855, volume 2, version 1.9. bpost: Documentation_Belgian_Template.pdf; MassPost guide January 2025, section 1. UPU country sheets (addressing unit, upu.int): France 09/2011, Belgium 09/2012, Netherlands 05/2016, Luxembourg 06/2013, Germany 01/2015, Austria 04/2014, Switzerland 06/2024, Liechtenstein 07/2002, Monaco 05/2004. Deutsche Post: Leitfaden Automationsfähige Briefsendungen 2024, section 2.2-2.4 (pp. 17-19). PostNL: the web page Adressering van buitenlandse zendingen (snippet only). Facts and short examples only, cited by page; the example addresses used as tests are in `testdata/addresses/authority-examples-tier2a.tsv`. # Address rules: the United Kingdom with Jersey, Guernsey and the Isle of Man, Ireland, Spain, Portugal, Italy, San Marino, the Vatican, Malta, Andorra How the rule parser (`Addresses/Tier2`) reads and standardizes these addresses. They are rows in the same tables as the first Tier 2 group (`Addresses/Data/T2/*.tsv`), read by the same reader; `docs/address-rules-tier2-a.md` describes the shared parts. This page lists what is particular to these countries. The sources are the UPU country sheets of the postal authorities; the authorities' own guides (Royal Mail's Programmers' Guide, An Post, Correos, CTT, Poste Italiane, MaltaPost) were not used, and no paid standard (BS 7666, UNE, UNI) was bought or read. Royal Mail's PAF data is not used. ## What the rules decide 1. **Last line.** The postcode and locality end the address. They come in two orders: - *postcode first*: Spain and Italy (five digits), San Marino (47890-47899), the Vatican (00120), Portugal (`1234-567`, a hyphen or en dash), Andorra (`AD500`): `28013 Madrid`, `2725-079 Mem Martins`, `AD700 Escaldes-Engordany`; - *locality first*: the United Kingdom, Jersey, Guernsey and the Isle of Man (six formats and `GIR 0AA`, UPU United Kingdom sheet), Ireland (Eircode: three characters, a space, four characters), Malta (three letters, four digits). The post town is on the line above the postcode or on the same line before it: `London SW1A 2AA`, `LONDON` / `SW1A 2AA`. A line that has digits, or starts a place inside a building, is not taken for a post town (Dublin postal districts, `DUBLIN 24`, are the exception); without a post town the address is refused. 2. **Admin area.** - Italy: a province code after the locality (`00122 ROMA RM`), checked against the 107 codes (the UPU Italy sheet lists 101; Caserta, Gorizia, Nuoro, Pordenone, Trieste and Udine are added from ISO 3166-2:IT). - Spain: the province on a line of its own after the postcode line, or in parentheses after the locality. Its name must be one of the Correos list (UPU Spain sheet, with the current co-official spellings) and the postcode must start with the province's own two digits; otherwise the line is not taken. - Ireland: the county, a line that starts `Co.` or `County` above the Eircode. 3. **Country.** From the country line, the postcode form that fits one country only (`SW1A`, Eircode, `AD500`, `1234-567`, `HMR 1428`, `00120`, `4789x`), a country letter before the postcode (`E-28013`, `I-00122`, `V-00120`, `SM-47890`), or a hint. The UK postcode area decides between the United Kingdom, Jersey (`JE`), Guernsey (`GY`) and the Isle of Man (`IM`); `CHANNEL ISLANDS` names Jersey or Guernsey, whichever the area says, and conflicts with any other area. Five digits are shared by France, Germany, Spain and Italy: the reader takes Spain only with a province line whose number is the postcode's prefix, Italy only with a province code, and does not read street words (`Plaza`, `Camino`, `Via`) as evidence, because American addresses use them too. Otherwise `address.country-undetermined`. 4. **Delivery lines.** - *Number first* (UK, Ireland, Malta; Andorra at the French post office): `49 Featherstone Street`, `38 Triq it-Tempji Neolitici`, `12 AVINGUDA TORRENT PREGO`. A line without a number is not a street line in the UK and Ireland (it may be a house name or a locality); in Malta and Andorra a line that starts with a road type (`Triq`, `Carrer`) is. - *Name then number* (Spain, Portugal, Italy, San Marino, Vatican, Andorra at the Spanish post office): the line starts with a road type (`Calle`, `Plaza`, `R`, `Av`, `Via`, `Viale`, `Piazza`, `Carrer`), then the name, then the house number, then staircase, floor and door. The house number is the first number after the name that leaves only floor and door words behind it, so `Calle 5 de Mayo 12` has number 12, and `Rua 25 de Abril` has no number. `s/n`, `sn` and `snc` stand for no number. Without a road type the line is left to the model. - *Boxes*: `P.O. Box` (UK, Ireland, Malta, Crown dependencies), `Apartado` (Spain, Portugal), `CCI` (Portugal), `Casella postale` (Italy, San Marino, Vatican), `Kaxxa Postali` (Malta), `BP`, `AP`, `Apartat` (Andorra). - Lines above the first delivery line are kept as written; lines after it (sub-locality, post office) are further delivery lines. In the UK and Ireland up to six lines without digits above the post town are accepted (house names, organizations), and nothing is warned about them. 5. **Everything else** is refused with `address.model-required`. Nothing is invented: no post town from a postcode, no county or province from a locality, no country from a street word. ## Postal and Display `Postal` follows each authority's own example lines; `Display` writes the same tokens in natural title case (`R Leal da Câmara 31 RC ESQ`; floor and door words are kept as written). | Country | Postal | Source | |---|---|---| | GB, JE, GG, IM | Lines as written; the post town in capitals; the postcode in capitals with one space, on a line of its own; country line `UNITED KINGDOM`, `CHANNEL ISLANDS` (Jersey, Guernsey) or `ISLE OF MAN`. No type is abbreviated: Royal Mail's free sheet publishes no abbreviation table. | UPU United Kingdom sheet 11/2011 pp. 1-2; Jersey, Guernsey, Isle of Man sheets 03/2005 | | IE | The addressee line as written; every other line (street, locality, county) in capitals; the Eircode on its own line, `T37 F8HK`; `CO.` kept. | UPU Ireland sheet 06/2017 p. 1-2 | | ES | Lines as written; the locality and the province in capitals; the commas before floor and door kept (`Calle Huertas 18, 4º, C`), none between street and number; the province on its own line. | UPU Spain sheet 01/2013 | | PT | Every line in capitals; accents kept; the postcode with a hyphen (`2725-079`). | UPU Portugal sheet 10/2024 ("address preferably printed in capitals") | | IT | Every line in capitals; no punctuation from the street line on (`VIA ROMA, 12` is `VIA ROMA 12`); a point or comma between digits stays (`KM 15,500`); the province code after the locality. | UPU Italy sheet 04/2020 | | SM, VA | Lines as written, locality in capitals; the Italian postcode system, no province code. | UPU San Marino sheet 10/2010, Vatican sheet 03/2005 | | MT | Lines as written; the locality in capitals; the postcode `HMR 1428` on its own line. | UPU Malta sheet | | AD | Lines as written; the locality in capitals; the postcode `AD500` (the letters are part of it) before the locality. | UPU Andorra sheet 09/2004 | The country line is printed only when asked. Country lines: `SPAIN`, `PORTUGAL`, `ITALY`, `IRELAND`, `MALTA`, `SAN MARINO`, `VATICAN`, `ANDORRA`. ## Where the sheets disagree or stop short - The UPU San Marino sheet prints `REP. OF SAN MARINO` in the example and `SAN MARINO` in the contact block; Postal prints `SAN MARINO`. The Vatican sheet prints `Città del Vaticano` in the example and capitals in the diagram; Postal prints capitals. - The UPU Andorra sheet prints `ANDORRE` for the Spanish post office; Postal prints `ANDORRA` for both. - The UPU Portugal sheet prints the postcode dash as an en dash in its text; Postal writes the hyphen the sheet shows in `2785-174`. - The Irish sheet's first example is in capitals for every line including the addressee, its second example only for the lines after the addressee; Postal leaves the addressee as written. - The UPU Italy sheet's list of provinces lacks six codes (see above); example `FRAZIONE MASSIGNANO SNC / 60129 MOSSO BI` pairs a postcode of Ancona with Biella; the reader does not look postcodes up, so it reads it. - The Spain sheet's province list uses older spellings (`LA CORUÑA`, `ORENSE`, `GUIPÚZCOA`, `BALEARES`, `GUADELAJARA`); both these and the current ones are accepted. - The UPU Guernsey sheet's contact block has no postcode (`St. Peter Port / GUERNSEY / CHANNEL ISLANDS`); it is not read. ## Not covered by rules (these go to the model) Addresses without a postcode; the UK county line when a county is written between the post town and the postcode (it would be taken for the post town); street lines without a road type in Spain, Portugal and Italy; Italian `P.zza`-style abbreviations that are not in the type table; Gibraltar and the other British territories (they are not in the group); personal postcodes of Malta that do not follow the format. ## Structural checks `PostalAddress.Validate` uses the structural layer's postcode patterns for each country (libaddressinput data); the Spanish province prefix and the Italian CAP ↔ province are not checked beyond what that data holds. The reader's own checks are the ones above: the UK area decides the dependency, a Spanish province line must agree with the postcode prefix, San Marino and the Vatican have their own postcode ranges. ## Keys and expansion Keys use the same segments as the first group (country, one segment per delivery line, locality, admin area, postcode) with the country's languages' dictionaries applied: Spanish, Catalan, Basque and Galician for Spain, Portuguese, Italian, Maltese with English, Irish with English, Catalan with Spanish and French for Andorra. A road type is the same word written in full or abbreviated (`C/` and `Calle`). `AddressExpander` takes `Spanish`, `Portuguese`, `Catalan`, `Basque`, `Galician`, `Maltese` and `Irish` besides the earlier languages; a street type leads the name in French, Italian, Spanish, Portuguese, Catalan, Basque, Galician and Maltese, so it expands in first position there. The words that libpostal lists as ambiguous in a new language are used only when that language is asked for, so the keys of the earlier countries are unchanged. ## Reading real-world text These rules were added to what the sheets show, so that real-world text is read: - English type last (`Barry Jackson Tower`, `High Street`) for GB, JE, GG, IM, IE and MT; the type's span excludes a closing period. - Unit lines: a prefix (`Flat 30`, `Interno 2`, `17º B`) or a unit alone on a line; unit tails split into designator and number. Italian unit lines are moved above the street line (Poste Italiane, Standard di composizione indirizzi 1.1). - Provinces: Italian and Spanish provinces in parentheses or after a comma, read without the parentheses; an Italian house-number letter (`25 D`); a province code with no locality (`33070 (pn)`) is refused rather than guessed. - Malta and Andorra read the number on either side of the name (`27 Andilina`, `Triq San Guzepp 27`); `C/name` written glued; `AD-500` is the postcode `AD500`; more Spanish, Italian, Portuguese and Andorran road types (CTT Anexo I abbreviations), `Apartado de Correos`, `CP`; more country names. - Britain and Ireland: when the post town is on a line of its own above a postcode line and another line without digits stands between it and the street, either could be the post town or a county, so the rules refuse (`address.model-required`) unless the post town is the only line in capitals. A post town on the postcode's own line (`Birmingham B6 5BA`) is never ambiguous. A nation or country name above the postcode (`ENGLAND`) is not a post town. - A hint of Italy keeps `97xxx` and `980xx` as Italian postcodes. ## Departures of Postal from the sheets (from the posts' own guides) Applying the post's own rule: (a) Portugal drops the closing period of an abbreviation (`R LEAL`, CTT Manual de Endereçamento p. 7, over the UPU sheet's `R. LEAL`); (d) Italy moves the unit line above the street and drops the parentheses around the province (Poste Italiane). Choices between two authorities, flagged: (b) Jersey locality printed as written (Jersey Post asks for capitals on the postcode only) while Guernsey, the Isle of Man and Britain print capitals; (c) Irish street lines left as written; (e) `PO Box` kept as the writer wrote it for GB, IE and MT (Royal Mail), where the UPU sheets print `P.O. Box`; (f) the Maltese postcode stays on its own line (UPU sheet), although the regulator's page writes locality and postcode on one line. ## Posts' own guides (retrieved 2026-11-06) Royal Mail PAF Programmers' Guide Ed. 7 v6 (e74e47fa…; guide only, no PAF data); Jersey Post addressing guidelines January 2025 (e2efb3d3…); Guernsey Post Guide to Services 2018/19 (8504e22a…); Poste Italiane, Standard di composizione indirizzi, August 2023 (5ac3c9f9…); CTT Manual de Endereçamento, October 2014, text copy (1eda0a44…; the official 2018 edition is image only); An Post sending guide (ab32456e…, no address examples); Malta Communications Authority page on addressing letters. Not used: Royal Mail's clear-addressing PDF (403), the Isle of Man Post page (JavaScript, no examples), Correos and MaltaPost (no guide found). The 54 examples taken from these guides are in `testdata/addresses/authority-examples-tier2b-posts.tsv`. ## Web forms (counties, island names, number markers, province lines) Read for web pages, from the Royal Mail PAF Programmers' Guide (the county is optional; a sub-building such as `Unit 4` comes before the building name), Jersey Post and Guernsey Post, the UPU Ireland sheet, CTT *Manual de Endereçamento* p. 7 (a number marker before the door number is unnecessary; lote and loja are units), Correos, Poste Italiane and the UPU Malta, San Marino and Andorra sheets: - A county, region or nation name on its own line, after `Co.` or `County`, or glued to the word (`Co.Louth`) is the admin area; the post town is the line above it. The county names are our own list (`t2-counties.tsv`). A bare county needs a post town and a delivery line before it, so `Dublin 8` is never a county. - A postcode written without its space (`CT161QD`) is split in the standard form. A one-line `14 Mill Lane Henley-on-Thames RG9 2AA` is split after the first road type. An Eircode alone ends an Irish address that has no post town. - Jersey, Guernsey and Isle of Man: the island name before the postcode is the country text and the parish or town above it is the locality; the printed layout stays as the sheets have it. `Jersey, Channel Islands` and `British Isles` are read through the postcode area. - `nº`, `n.º`, `Nr` and `Nº` before a number are dropped from the standard form; `CL.MAYOR` is split into type and name; Catalan ordinals (`3r 2a`), `baixos`, `bis`, `ter`, `km 5`, `nave`, `parcela`, `lote`, `loja`, polígono and zona industrial types, and the Basque `Name Kalea 5` order are read. - A country in parentheses after the locality (`(Italy)`, `(RSM)`, `(AD)`) is the country text. `AD-500` is read as AD500. `80134 Napoli, NA` and `Napoli NA` give the province; an Italian address with a province code needs no postcode. Not read yet, and refused to the model: `28013 Madrid, Calle Mayor 5` (postcode first), an untyped Spanish street (`Nagusia 5`), and an Italian `Via` address with no hint and no province code (shared with Catalan). ## Sources UPU country sheets (addressing unit, upu.int, `PostalEntitiesFiles/addressingUnit/En.pdf`): United Kingdom 11/2011, Jersey 03/2005, Guernsey 03/2005, Isle of Man 03/2005, Ireland 06/2017, Spain 01/2013, Portugal 10/2024, Italy 04/2020, San Marino 10/2010, Vatican 03/2005, Malta, Andorra 09/2004. Facts and short examples only, cited by sheet and page; the examples used as tests are in `testdata/addresses/authority-examples-tier2b.tsv`. The libpostal dictionaries of `es`, `pt`, `ca`, `eu`, `gl`, `mt` and `ga` at the pinned tag, through the ingest tool. ## Island lines without commas Jersey, Guernsey and the Isle of Man are also written on one line with no commas: `Victoria Road Douglas IM1 1AA`, `Tromode Road Douglas Isle of Man IM2 5AA`, `Hill Street Jersey JE4 7BR`, `Douglas Isle of Man IM1 2AB`. The street ends at the last road type that leaves one to three words for the post town (Royal Mail PAF Programmers' Guide: the thoroughfare, then the post town); the island's name after the town is the country text, and when only the island name follows the road it is read as the town. Nothing is added to the input. ## Country text beside the island and Irish postcodes - `IOM` (and `IoM`) after an Isle of Man postcode is the island's name as the Manx pages abbreviate it; `(UK)` beside a British, Manx, Jersey or Guernsey postcode names no country the postcode does not already name, so the postcode area decides (Royal Mail PAF Programmers' Guide and the UPU Isle of Man, Jersey and Guernsey sheets give the postcode areas). `Douglas, Isle of Man (UK) IM1 1AA` keeps `Isle of Man (UK)` as the country text. - `Jersey, Channel Islands` before a Jersey or Guernsey postcode is read as the island (UPU Jersey and Guernsey sheets print the island above the postcode). - A British nation or country (`England`) on a unit of its own above the postcode is the country when a county stands above it (`Guildford, Surrey, England, GU1 3AA`); with no county above it the nation stays the admin area, as Jersey Post prints it (`Bournemouth / ENGLAND / BH1 1AA`). - A Jersey or Guernsey line that holds a house name, a road and the parish with commas above the island name is cut before its last piece when that piece is a place with no digits; `Mon Plaisir, La Rue du Hocq, St Clement` above `Jersey JE2 6LF` gives the locality `St Clement` (Jersey Post prints the parish on its own line). A line that ends in a street and number is not cut. - An Irish text with no postcode is recognised without a hint only when it ends in `Co.` or `County` and a county of the Republic, or in a Dublin postal district (`Dublin 8`; UPU Ireland sheet: `CO. CORK`, `DUBLIN 24`). A bare town name never is, and `CO` without a period is left to the United States (Colorado). # Address rules: Denmark, Sweden, Finland, Norway, Iceland, Estonia, Latvia, Lithuania How the rule parser (`Addresses/Tier2`) reads and standardizes these addresses. They are rows in tables of their own (`Addresses/Data/T2/*-north.tsv`), read by the same reader as the other Tier 2 countries; `docs/address-rules-tier2-a.md` and `-b.md` describe the shared parts. This page lists what is particular to these eight. The sources are the posts' free public guidance where we found it, and the UPU country sheets for the rest (below). No paid standard (DS, SS, SFS, NS, LVS, LST) was bought or read. ## What the rules decide 1. **Last line.** The postcode and the locality end the address. The country decides the form: - *four digits, left of the locality*: Denmark (`8660 SKANDERBORG`) and Norway (`0025 OSLO`); - *three digits*: Iceland (`220 HAFNARFIRĐI`); - *five digits with a space after the third*: Sweden (`123 45 LILLBYN`, the joined `12345` is read and written with the space); - *five digits*: Finland, Estonia (`11212 TALLINN`; in the country-side form the locality is the region, `69501 VILJANDIMAA`) and Lithuania (`LT-04340 Vilnius`); - *locality first*: Latvia, `RIGA, LV-1050`. The comma and the postcode may be on the line below the locality (`SALACGRIVAS NOV.,` / `LV–4035`). A country prefix before the postcode (`DK-`, `SE-`, `S-`, `FI-`, `NO-`, `N-`, `IS-`, `EE-`, `LV-`, `LT-`, also with an en or em dash) names the country. 2. **Country.** From the country name (English, the country's own language, French, German: `DENMARK`, `DANMARK`, `SVERIGE`, `SUOMI`, `NORGE`, `ISLAND`, `EESTI`, `LATVIJA`, `LIETUVA` …), the prefix, a hint, or `LV-` (the only form of its kind). The digits alone never decide: five digits belong to Finland, Estonia, Lithuania, Sweden, France, Germany, Spain and Italy; four to Denmark, Norway, Belgium, Austria, Switzerland, Liechtenstein, Luxembourg; and three digits are a postcode only when Iceland is known. Without evidence the result is `address.country-undetermined`. With a named or hinted Finland, Estonia, Lithuania or Sweden, five digits are that country's even where another form would use them (`00120` is Helsinki, not the Vatican; `97xxx` and `98xxx` exist in Finland and Estonia). 3. **Delivery lines.** - *Street line*: the street name, then the house number, then floor, door or apartment. The name has no required type word (`Kastanievej`, `Karl Johansgate`, `Laisvės pr.`, `Kr. Barona street`). The number may carry a letter (`25 B`, `10a`) and an apartment after a dash (`6–1`, `40-12`, `90—23`), a slash for a double number or building (`48/50`, `45/4—22`). - *Floor and door*: after the number, written as the writer set them: `15, 2` and `34, 1. th` and `34, st. a19` (Danmarks Adresser), `25 B 13` and `4 A 16 Ullakkohuoneisto` (UPU Finland), `lgh 1234` (PostNord), `7, dz. 1` (Latvian rules). Commas the writer set are kept; no period, `sal` or door letter is added or changed. - *A supplementary town name* after the last comma of a street line (`Kastanievej 15, 2, Agerskov`) stays on the line; on one-line input it becomes the next delivery line. - *Boxes*: `Postboks` (DK, NO), `PO Box` (NO), `BOX` (SE), `PL`, `PB`, `P.O.Box` (FI), `Pósthólf` (IS), `P. d.` (LT). A name after the number (`Postboks 55 Falkum`, a box facility) stays on the line. - *Poste restante*: `Poste restante` (FI, NO), `Nõudmiseni` (EE), `Iki pareikalavimo` (LT). - Lines above the first delivery line are kept as written (addressee, organization, `Att:`, `c/o`, `v/`, `Bolignummer H0302`, a farm name). A line that starts with `c/o`, `Bolignummer` or `Bnr` is never the street line. - *No delivery line at all* is an address: a farm, a village or a place name stands in for the street in Norway (`Vikane`), Estonia (`Allika talu`, `Halliste alevik`) and Lithuania (`Galinių k.`, `Dubingių pšt.`), and a recipient with the postcode alone is a Norwegian address (`9672 INGØY`). Up to six name lines without digits are taken, with no warning. 4. **Everything else** is refused with `address.model-required`. Nothing is invented: no town from a postcode, no street type, no floor, no country from digits. ## Postal and Display `Postal` follows each sheet's own example lines; `Display` writes the same tokens in natural title case (`123 45 Lillbyn`, `Nyby 10`; an all-capital organization suffix such as `AB`, `AS`, `Oy`, `PL`, `PO` stays). | Country | Postal | Source | |---|---|---| | DK | Lines as written (street, number, floor, side and supplementary town name stay as set); the locality in capitals; box line `Postboks 321`; postcode without country code. | UPU Denmark 05/2024; Danmarks Adresser, *Gengivelse af adresse* A.1-A.4 | | SE | Lines as written; locality in capitals; `123 45`; box line `BOX 1121`; the apartment number after the street on the same line (`lgh 1234`). | UPU Sweden 11/2024; PostNord, *Adressering* | | FI | Lines as written; locality in capitals; `PL 900`, `P.O.Box 102`; `c/o` above the street line. | UPU Finland 11/2024 | | NO | Lines as written; locality in capitals; `Postboks 1181`, `PO Box 15`. | UPU Norway 01/2013; Posten Norge, *Adressering* | | IS | Lines as written; locality in capitals; `Pósthólf 1550`. | UPU Iceland 07/2002 | | EE | Lines as written; locality or region in capitals; the house and apartment number as written (`6–1`). | UPU Estonia 06/2014 | | LV | Lines as written; the locality in capitals, a comma, then `LV-` and the four digits on one line. | UPU Latvia 09/2023; Latvian rules for unified postal addresses (Latvijas Vēstnesis, id 51772) | | LT | Lines as written; `LT-` and the five digits, then the locality as written (the sheet prints `Vilnius` and `Kaunas` in mixed case and province names in capitals). | UPU Lithuania 06/2018; Lietuvos paštas, *Siuntos* (post.lt) | The country line (English, in capitals for Postal) is printed only when asked: `DENMARK`, `SWEDEN`, `FINLAND`, `NORWAY`, `ICELAND`, `ESTONIA`, `LATVIA`, `LITHUANIA`. ## Where the sources disagree or stop short - **Locality case.** The UPU sheets print the locality in capitals; Posten Norge's own page and the Danish address register print it in mixed case (`0161 Oslo`, `1553 København V`). `Postal` follows the UPU sheets; the mixed-case form is the `Display` rendering, and the tests pin it so. Lithuania is the one sheet that prints the locality in mixed case, and its Postal keeps it. - **Dash.** Latvia and Lithuania print `LV–1050`, `LT–04340` with an en dash. `Postal` writes the hyphen (as the Portuguese postcode does). The house and apartment dash is kept as written. - **Latvian locality line.** The sheet prints `SALACGRIVAS NOV.,` and `LV–4035` on two lines and `RIGA, LV–1050` on one; `Postal` writes both on one line, which is what the sheet's rule says (the postcode right of the locality, separated by a comma). The trailing comma after a Latvian sub-locality (`Valdemāra street 42, Ainaži,`) is dropped. - **The Latvian street line** is two lines in the sheet's diagram (`Kr. Barona street` / `7, dz. 1`); the extracted text of the sheet suggests one wrapped line, which is how the test reads it. - **Foreign addresses on Posten Norge's page** (`DK-5000 Odense`, `IS-107 Reykjavik`, `SE-582 20 Linköping`) are Norwegian-post renderings of other countries' addresses; they are read (country, postcode, locality) and not compared as Postal examples, because each country's own sheet decides its Postal form. - **Not read by the rules, so refused:** the Latvia Post contact block (`Ziemeļu iela 10. Lidosta “Rīga”`: a place name after the number) and Latvian post-office suffixes such as `Rīga–50`; the model decides those. The Iceland sheet's contact line is transcribed as the PDF text gives it (`Stórhöfð1 29`, which looks like an extraction artifact of `Stórhöfði 29`). - **Posti** (Finland), **Omniva** (Estonia), **Latvijas Pasts** and **Íslandspóstur** publish no free addressing page we could find with examples of their own; their UPU sheets stand in. Finland's language-board guidance (Kielitoimiston ohjepankki) says the same about the postcode and locality but is not a post's guide and is not used for examples. - The Danish registry's `Postal` conventions (`th.`, `tv.` with a period; floors with a period) are written as the writer wrote them; nothing is added or rewritten. - Norwegian Nynorsk has no libpostal dictionary of its own; the Bokmål one (`nb`) covers both. ## Expansions `ExpansionLanguages.Danish`, `Swedish`, `Finnish`, `Norwegian`, `Icelandic`, `Estonian`, `Latvian` and `Lithuanian` use the libpostal dictionaries `da`, `sv`, `fi`, `nb`, `is`, `et`, `lv`, `lt` (street types, directionals, unit and level words, post office words, ambiguous expansions) at the pinned tag, listed in `upstream/libpostal-dictionaries/PIN`. Each language keeps its own ambiguous-expansion list. The German and Dutch ending expansion (`Hauptstr` → `Hauptstrasse`) is not applied to the northern languages. Keys of the United States, Canada and the first two groups do not change. ## Labels and extra forms (recovery round) These forms are read by rules: - **Country code before the postcode** (`DK-1352`, `NO-6419`, `SE-839 89`, `IS-102`, `FI-02970`, `EE-`): the code is the `country` field and the postcode is the digits after the dash. Latvian and Lithuanian postcodes keep `LV-` and `LT-` as part of the postcode. - **Unit labels.** Floor and door as the writer set them (`1. th`, `st. a19`, `13`, a Norwegian dwelling number `H0219`) are one unit number; a designator word (`lgh`, `dz.`, `korp.`, Estonian `korter`) is the designator and what follows it the number. In Estonia, Latvia and Lithuania the apartment after the house number's dash (`6–1`, `2 - 22`, and in Estonia also a slash, `46/49`) is the unit number; a Latvian building (`5 k-8`) stays part of the house number. The dash keeps its spacing in Postal. - **Box words**: `Pb.` (DK, NO), `Postbox` (SE), `Postilokero` (FI), `Pósth.` (IS), `Postkast`, `PK` (EE), `Pasta kastīte`, `PK`, `p/k` (LV). - **A company name that runs into the street on one line** is split after its legal form (`ApS`, `A/S`, `AB`, `Oy`, `AS`, `ehf`, …) when the rest is a street line. A person's name that runs into the street is not split: where the name ends and the street begins is not certain, so such a line is read as a street name or refused, and the model decides. - **Latvia.** `Valmiera, Valmieras nov., LV-4201`: the municipality (`nov.`) is the admin area and the town the locality; a parish (`pag.`) line or segment is the dependent locality, and with a parish and no street the line above it is the house name (the Latvian rules: the house name stands where the street would in the countryside). A parish with the house name in front of it on one line (`Saulītes Sēļu pag. Sēļi`) is split into house name, parish and locality. - **Estonia.** `48103 Põltsamaa Jõgeva maakond`: the county is the admin area; a `vald`, `linn` or `linnaosa` line is the dependent locality; with such a line (or a village locality and a line besides the addressee) and no street, the line above is the farm or house name. What the rules leave to the model on purpose: a lone name line above a postcode in Iceland, Norway, Finland and Estonia (recipient or farm name: the sheets do not say), a Latvian `Name 8` without `iela` (a street or a farm, the Latvian rules allow both), a person's name glued to a street, a street and a locality on one line (`Marijas iela 22 Rūjiena`), and a postcode written without its `LV-`. ### Second recovery round: Lithuania - **Village and municipality.** The UPU Lithuania sheet prints the rural address with the municipality (`RASEINIU R. SAV.`) and the locality apart. A line (or comma segment, or the words after the house number on one line) ending in `k.` (kaimas) is the dependent locality, including `Dabužių I k.`; one ending in `sav.` (`r. sav.`, `m. sav.`) is the admin area. `Pienionių k., Anykščių r. sav.` in one line of a label is two lines. The municipality stays where the writer put it in the printed layout; only its field changes. - **Box.** `PD 2876` and `Pašto dėžutė 12` as well as the sheet's `P. d.`. - **Apartment.** `butas 1` and `b. 3` after the house number are the designator and the unit number. A slash between house and apartment (`66/72`) is read like the sheet's dash, printed as written. - **Company name.** `MB` after a comma (`Sakalauskas ir Ko, MB, Aušros g. 23`) belongs to the name before it; as `UAB` and `AB` it also ends a company name that runs into the street on one line. ## Country inference (no hint) Without a country hint the reader first tries the plain Tier 2 read. When that fails as `country-undetermined` or `no-last-line`, and the text has a digit, is at most 600 characters and shows some evidence of a language, it is read once as each Tier 2 country (a trial read with the candidate as the hint). The country is chosen only if: 1. exactly one country among those that share the postcode shape has evidence in the text, and 2. that country's trial read succeeds. The result then carries confidence 0.97 and the `CountryInferred` flag. Otherwise the text is refused as before; nothing is guessed. **Evidence** (data in `Data/T2/t2-inference.tsv`): letters at or above U+0100 and `ß` (Latin-1 letters never count, since names carry them); the country's street and place words and suffixes; an abbreviation directly before a house number of up to four digits (`g.`, `pr.`, `al.`, `tn`, `mnt`, `pst`, `Top`); an ordinal before a floor word (`3. sal`); Liechtenstein's municipality names; and, for the Czech Republic and Slovakia, the postcode's first digit (1-7 Czech, 0 and 8-9 Slovak; a leading 0 counts only when written `NNN NN`). **Vetoes** stop an inference even when the evidence is unique: the text ends in another country's name, a zone of another country, or an Italian province code left in the locality; a Dutch-looking `dddd LL`; an unspaced five-digit postcode in Swedish form (it may be Finnish with Swedish words); letters outside the winner's alphabet. **Refused by design.** German-language text in Austria, Switzerland, Belgium and Luxembourg without a country name or distinctive word; Czech and Slovak ASCII text whose postcode falls in no zone; Danish and Norwegian text built from shared words (`Postboks ... Oslo`); lines without a postcode (`Verslo g. 5, Vilnius`), which count as evidence only. Apart from the short evidence lists of Liechtenstein municipalities and Estonian towns and counties, no gazetteer of places ships. ## A person written in front of the street Denmark, Norway, Sweden and Finland write a street name as one word or end it in a type word (`Karl Johans gate`, `Ole Bulls plass`; Posten Norge, *Adressering*; PostNord; Danmarks Adresser, *Gengivelse af adresse*). Exactly three words before the house number that start with a given name and do not end in a type word (`Nils Larsen Nestangen 58`) are a person written in front of a one-word street with no comma or line break; longer names are read as before, since a four-word street name is as likely as a person with a middle name. Where the person ends cannot be told from the words, so the rules refuse the line and the model reads it; they used to read the person as part of the street name. The given names are our own list (`docs/names.md`). ## The Estonian county beside the town Omniva (*Kuidas kirja adresseerida?*) writes the postcode, the town and then the county (`maakond`). Web pages also put the county on a line of its own after the town (`Pärnu mnt 12, 10148 Tallinn, Harju maakond`, `Pärnu mnt 12, Tallinn 10148, Harju maakond`), or between the town and the postcode (`Ülikooli 18, Tartu, Tartu maakond, 51003`, `Tallinn, Harju maakond 10148`). A county line after the postcode line is joined to it, and a county that stands between a plain town word and the postcode makes that word the locality and the county the admin area. A village, rural municipality or town-status word (`küla`, `vald`, `linn`, `alevik`) before the county keeps the reading above. The postcode may carry the country letters (`EE-80010`; UPU Estonia sheet). ## Estonia without a hint, and without a postcode - **Towns and counties are evidence of the country.** The towns (`linn`) of Estonia and the one-word county names (`Harjumaa`, `Tartumaa`) are listed as places in `Data/T2/t2-inference.tsv`, as Liechtenstein's municipalities are. A five-digit text with a street, a postcode and one of these towns (`Pikk 5, 10133 Tallinn`) is read as Estonian when no other country that can read it has evidence of its own. Towns whose name is also an ordinary word in a neighbouring language (`Elva`, `Turi`, `Valga`, `Pussi`, `Polva`, `Rapina`) are left out, so a place word never decides against a country that writes it. `põik` (lane), `väljak` (square) and `postkast` (post office box) are street and delivery words of Omniva. - **`allee` is Estonian as well as French, German and Dutch** (`Pärna allee`): without that row the language check refused every Estonian avenue. - **A last line with no postcode.** Web pages often leave the postcode out. A town of Estonia alone, or with its county, on the last line after a delivery line is the locality (and admin area): `Pärnu mnt 12, Tallinn`, `Pärnu mnt 12, Tallinn, Harju maakond`, `Aia 12, Tartu, Tartu maakond`, and with `Estonia` after it. The postcode is left empty. The town must be on the list; a bare name is never taken for a locality (`Pikk 5, Springfield` stays refused). Only when Estonia is hinted, named, or the sole country with evidence. Measurements are in `docs/maintainers/address-recovery.md`. # Address rules: Poland, the Czech Republic, Slovakia, Slovenia, Croatia, Hungary, Romania, Mexico, Australia How the rule parser (`Addresses/Tier2`) reads and standardizes these addresses. They are rows in the same tables as the earlier Tier 2 groups (`Addresses/Data/T2/*-c-south.tsv`, read next to the base files), read by the same reader (`Tier2Reader.Central.cs` holds what is particular to this group); `docs/address-rules-tier2-a.md` and `-b.md` describe the shared parts. ## What the rules decide 1. **Last line.** Postcode and locality end the address. - *Postcode first*: Poland `00-950 WARSZAWA` (five digits with a hyphen), Czech Republic and Slovakia `190 16 PRAHA 916` / `921 01 PIEŠŤANY` (three digits, a space, two digits), Slovenia `1000 Ljubljana` and Croatia `10000 Zagreb` (four and five digits), Romania `200716 CRAIOVA` (six digits), Mexico (five digits, locality and state abbreviation), and Hungary in two orders (below). - *Hungary*: the authority order puts the locality on the line above the street (`BUDAPEST` / `Kossuth Lajos utca 12.` / `1037`), and the common "1037 Budapest, Kossuth Lajos utca 12." order is accepted too. Postal always prints the authority order. - *Australia*: `RICHMOND VIC 3121`, locality, state or territory (`NSW VIC QLD WA SA TAS ACT NT`, capitals only) and four digits on one line. - *Mexico*: a state abbreviation (`CDMX`, `JAL`, `NL`, ...) or the colonia line is what makes five digits Mexican; the state is one of the 32 SEPOMEX abbreviations. 2. **Shared digit forms.** Five digits fit Mexico and Croatia, four fit Slovenia and Hungary, three plus two fit the Czech Republic and Slovakia. Without a country line, a hint or a postcode prefix that decides, the reader takes a country only on evidence that belongs to one of them: a state abbreviation or colonia line (MX), Croatian or Slovenian street words, a Hungarian street type. Czech versus Slovak needs the country line or a hint. 3. **Country.** From the country line (in English, the local language or the sheet's abbreviation such as `CZECH REP.`), the postcode form that fits one country only (`00-950`, six digits, an Australian state line), or a hint. 4. **Delivery lines.** - *Type, name, number* (Poland `ul. Marszałkowska 12/4`, Czech Republic `Valdštejnské náměstí 17/4`, Slovakia, Slovenia `Cesta v Mestni log 1`, Croatia `Ilica 12`, Hungary `Kossuth Lajos utca 12.`, Romania `Str. Bariera Valcii, nr. 215`): the street type is part of the street name or leads it per country; `č. p.` (Czech house number without a street), `nr.` and `Nr` markers are kept as written. A bare house number with `/` (orientation/descriptive) is a house number. - *Number first* (Australia): `12 Smith St`, `Unit 4, 12 Smith St`, `4/12 Smith St`, `Lot 16 Godfrey Ave`, `Level 2, 100 Pitt St`. The unit designator is abbreviated as Australia Post asks (`Unit`, `Lvl`, `Apt`); a street name's own words (`St Kilda Rd`, `McKillop St`, a trailing `West`/`W`) are not changed. - *Boxes*: `skr. poczt. Nr 115` (PL), `poštovní přihrádka 87` (CZ), `P.O. Box`/`PO Box`, `Apartado Postal` (MX), `Locked Bag`, `GPO Box`, `Private Bag`, `Reply Paid` (AU), `poste restante`, `dodávací schrána B/52` (CZ delivery box). The box number may carry `-` or `/` (`70-123`, `B/52`). - Lines above the first delivery line (titles, addressee, `do rukou p. Ptáčka`, `nar. 1. 4. 1946`) are kept as written. 5. **Postcode ↔ region.** The reader does not duplicate ranges. `Validate()` uses the structural layer's postcode patterns, so a Mexican prefix ↔ state or Australian postcode ↔ state check is exactly what libaddressinput's zip rules for the state hold; the reader itself checks only that an Australian state is one of the eight and in capitals. 6. **Everything else** is refused with `address.model-required`. Nothing is invented: no locality from a postcode, no state from a locality, no country from a street word. ## Postal and Display | Country | Postal | Source | |---|---|---| | PL | Locality in capitals, postcode `00-950` before it; lines above as written; country line `POLAND` | Poczta Polska guide (own); UPU Poland 08/2018 | | CZ, SK | Locality in capitals, two spaces after the postcode (`190 16 PRAHA 916`); other lines as written; `Poste restante` as written | Česká pošta (own); UPU Czech Republic 02/2018, Slovakia 01/2013 | | SI | Postcode, locality in capitals | UPU Slovenia 05/2016 | | HR | Postcode, locality in capitals | UPU Croatia 05/2014 | | HU | Locality in capitals on its own line above the street; postcode on the last line | Magyar Posta guide (own); UPU Hungary 01/2012 | | RO | Street line as written, postcode and locality in capitals | UPU Romania 10/2007 | | MX | Street, colonia, `CP` line with the locality and state abbreviation, capitals as the sheet prints | UPU Mexico 03/2017 | | AU | Street line, unit and designators in capitals without periods; `LOCALITY STATE 3000` on one line with single spaces | Australia Post "Correct addressing" (2022), Address Presentation Standards (1999); UPU Australia 07/2017 | `Display` writes the same tokens in title case with the authority's abbreviations (`Sc Pasaj SA`, `Praha 916`, `Motosport, a. s.`); identifiers with digits keep their capitals; particles stay lower case (`Dvory nad Lužnicí`); a legal-form abbreviation and Australian `McKillop`, `St Kilda` keep their shapes. The country line is printed only when asked. ## Where the guides disagree or stop short - Posts' own guides were found for Poland, the Czech Republic, Hungary and Australia only. For Slovakia, Slovenia, Croatia, Romania and Mexico the UPU sheets stand in. The Polish and Hungarian guides show their examples as images, so only their rules are read; the examples used as tests come from the UPU sheets. - The sheets print the country as `POLSKA`/`POLAND` and `CZECH REP.`; Postal prints the English country name (`POLAND`, `CZECH REPUBLIC`). - Polish postcodes are written with a hyphen where the sheet shows an en dash. - The Czech and Slovak sheets print the locality in mixed case in some examples while the post's own rules say capitals; Postal uses capitals. Czech Post requires two spaces after the postcode (`190 16 PRAHA 916`), the UPU text only recommends it; Postal prints two. - The Slovak sheet prints `POSTE RESTANTE` in capitals; Postal prints `Poste restante` as Czech Post does. - The 1999 Australia Post standards print a street line with a period (`17 Jones St.`) in the UPU example and ask for capitals and no punctuation elsewhere; Postal uses capitals without the period. The standards prefer two spaces between locality, state and postcode, the 2022 brochure prints one; Postal prints one. The 1999 text cites AS 4590 as its own source; the standard itself was not read. - Typos in the printed text: the Slovenian sheet's `Prešemova` is kept as printed; the brochure's `PARKVILLE VIC 305 2` is not transcribed. - Some AU element examples have no complete address; the test appends a last line and, where needed, a house number of our own. ## Departures of Postal from the printed text Each applies the country's own postcode or name form, a post's own written rule, or the never-invent rule. 1. Country line in English (`POLAND`, `CZECH REPUBLIC`): the country's English name form. 2. Polish postcode hyphen for the sheet's en dash: the postcode's own form. 3. Two spaces after the Czech/Slovak postcode where the UPU example shows one: Česká pošta's written rule. 4. Locality in capitals where the UPU examples for PL, CZ and SK print mixed case: the posts' own written rule. 5. `Poste restante` as written, not the Slovak sheet's capitals: Česká pošta's usage. 6. AU street line in capitals without the period (`17 JONES ST`): Australia Post's written rule. 7. AU one space between locality, state and postcode: the 2022 brochure's own print. ## Keys and expansion Keys use the same segments as the earlier groups with each country's dictionaries applied: Polish, Czech, Slovak, Slovenian, Croatian, Hungarian and Romanian from libpostal's `pl`, `cs`, `sk`, `sl`, `hr`, `hu` and `ro`; Spanish for Mexico, English for Australia. `AddressExpander` takes the seven new languages (`ExpansionLanguages.Polish` … `Romanian`); a leading street type expands in first position in Polish, Czech, Slovak, Slovenian, Croatian and Romanian. House-number markers (`Nr`, `nr.`, `č. p.`) are skipped in the key and the Polish box words are normalized. The keys of US/CA and the first two Tier 2 groups do not change. ## Reading real-world text - **Romania.** The postcode may come last (`Pitești, 110040`; `București, Sector 3, 032391`; `Brașov, RO-500483`), and a sector or county after the locality (`011371 București, Sector 1`; `400241 Cluj-Napoca, județul Cluj`), which on one line is a unit of its own after the postcode, is joined to it. A county (`jud.`, `județ`, `județul` and a name) is the admin area, kept as written and printed after the locality, `400241 CLUJ-NAPOCA, JUDEȚUL CLUJ`. A sector stays part of the locality. The comma-below letters `ș` and `ț` now fold to `s` and `t` like the cedilla forms (the fold missed them, so Romanian words never matched their keys). - **Australia.** The state in full is read and standardized to its abbreviation (`Camira Queensland 4300` is `CAMIRA QLD 4300`); a comma before the state (`Downer, ACT 2602`) is read on one line; a lowercase state abbreviation is read once the country is known to be Australian (without a hint `Sydney nsw 2000` is still refused). On one line without separators the locality starts after the only street type that can end a street name; where two could, the text is refused. `4/12 Smith St` is a unit number and a street number that print as written, in two parts. - Slash numbers in Croatia and the Czech Republic (`7/6`, `320/62`) stay house numbers: the slash is not always a unit. Without a hint the four-digit (SI) and `NNN NN` (SK) forms are shared with other countries and stay `address.country-undetermined` unless country inference finds evidence (SI and SK now read about half of the register text without a hint). A refusal as `no-last-line` is mostly text without a postcode, which the rules never invent. What the rules read now, each from the guide named: - **Poland.** Five digits without the dash (`08530 Dęblin`) are `08-530` when the text is known to be Polish (a hint or the country name); without that they are not read as Polish (France, Germany and others share the form). Poczta Polska, *Zasady poprawnego adresowania*, 1.2: the building number and, after it, the number of the flat or premises. `9/93` is a house number `9` and a unit number `93`, printed together as written; `m. 22` (mieszkanie) and `lok. 5` (lokal) are the designator and the unit number, also when a comma set them apart on one line; `Skrytka 396` is a box. - **Slovenia.** A single letter after a space belongs to the house number (`213 a`, `4 c`; the UPU Slovenia sheet writes `p.p.` and the letter forms follow Slovenian usage); `p. p. 3795` is a box. - **Slovakia.** `P.O.BOX 341` and `Poštový priečinok 1084` start a box line (UPU Slovakia sheet: P.O. Box). A box word inside a street line (`Závodie P.O.BOX 1970`) leaves the line to the model instead of reading it as a street name. A legal form after a comma (`Mikuš a spol., v.o.s., Neded 931`) belongs to the company name before it. - **Mexico** (UPU Mexico sheet: colonia line, postcode, locality and state abbreviation): - `C.P.` / `CP` (código postal) stands in front of the digits and is part of the postal-code field as the writer set it; the standard value is the five digits. The code may come after the locality (`León, C.P. 37240`; `…, La Barca, Jalisco, C.P. 47912`) or before it with the locality and state after (`C.P. 38500, Apaseo el Alto, Guanajuato`). The word alone does not make the text Mexican: without a hint, a Mexican state or a colonia line is needed (Spain writes `C.P.` too). - A line that starts with `Col.`, `Colonia`, `Fracc.`, `Fraccionamiento`, `Barrio`, `Pueblo`, `Ejido`, `Residencial` or `Unidad Habitacional` is the dependent locality; a colonia that follows the street number on the same line (`… 105B Col. Los Angeles 27140 Torreón`) is a line of its own. A quarter named without such a word stays an additional line as written. - `S/N` (sin número) is the house number of a street without one. `Ciudad de México` and `Estado de México` are states, not the country; `Coahuila de Zaragoza` and `Michoacán de Ocampo` are read. A state name after `de` (`Tepatitlán de Morelos`) is part of the town. When the country stands on a line of its own, a `México` left on the locality line is the state. A lone `México` that could be the state or the country (`…, Ecatepec de Morelos, México, C.P. 55418`) is left to the model. - A legal form after a comma (`Servicios Díaz, S.C., Calle …`) belongs to the company name. ### Web forms of Hungary, Romania and Australia Forms people write on web pages, read from the posts' layouts above (all examples in the tests are our own): - **Hungary.** Postcode, locality and street on one line without commas (`1068 Budapest Király utca 80.`): the locality is the word after the postcode, with a district (`VI. ker.`) if one follows; what is left must read as a street line with a Hungarian street type or a box. A locality of several words is not guessed, and a street line that comes before the postcode with the locality in front of it (`Pécsudvard Arany köz 98. 7762`) stays with the model, because a recipient's name cannot be told from a locality word. A box number may end in a period (`Pf. 12.`). Without a hint, `utca` and the full word `postafiók` are Hungarian evidence (`Pf.` is not: German-speaking posts abbreviate Postfach the same way). - **Romania.** The label of the postcode (`cod`, `cod poștal`, `Cod poștal:`) is left out of every field. A postcode written between the street and the locality is joined to the locality after it. A locality written on the street line without a comma is the plain words after the last number (`Strada Frumoasă Nr. 51 București, Sector 1, 010986`); a tail with a period or a building word (`bl.`, `ap.`) is never taken for one. A text with no locality or no postcode still goes to the model. - **Australia.** Commas around the state and the postcode (`Sydney, NSW, 2000`; `Sydney NSW, 2000`) are read like the unseparated form, provided a street or box line and a locality stand before the state. `Unit 3/12 Smith St` and `Shop 4/100 George St` keep the unit and the street number apart. - **Names.** `adw.` (adwokat), `r.pr.` (radca prawny), `radca prawny`, `adwokat` and `nadzw.` are Polish titles alongside `dr`, `dr hab.`, `mgr`, `inż.` and `prof.`. `adw.` and `r.pr.` count without a language hint; the full words need Polish. No gender or salutation is inferred. ## Sources UPU country sheets (`upu.int/UPU/media/upu/PostalEntitiesFiles/addressingUnit/En.pdf`, retrieved 2026-11-05): Poland 08/2018, Czech Republic 02/2018, Slovakia 01/2013, Slovenia 05/2016, Croatia 05/2014, Hungary 01/2012, Romania 10/2007 (`rouEn`), Mexico 03/2017, Australia 07/2017. Own guides: Poczta Polska "Zasady poprawnego adresowania i oznakowania przesyłek listowych"; Česká pošta "How to correctly label the consignment" (about 19 recommended formats); Magyar Posta "Tájékoztató a levélküldemények helyes címzéséről"; Australia Post "Correct addressing" (November 2022) and "Address Presentation Standards" (1999). No licence is stated on the guides; facts and short examples only, cited by page. The examples used as tests are the 89 rows of `testdata/addresses/authority-examples-tier2c-south.tsv`: UPU Poland 4, Czech Republic 9, Slovakia 8, Slovenia 3, Croatia 3, Hungary 2, Romania 3, Mexico 2, Australia 6; Česká pošta 19; Australia Post 30 (1999 standards and 2022 brochure, element examples included). The libpostal dictionaries of `pl`, `cs`, `sk`, `sl`, `hr`, `hu` and `ro` at the pinned tag, through the ingest tool. ### Settlement lines, Bucharest sectors and addressees written into the street (round three) - **Czech Republic, Slovakia, Croatia: a settlement line.** A line of plain words (one to four words, no digit, no closing period, no single letter, no unit, box or street-type word) between the street line and the postcode line names the part of the municipality or the settlement the street lies in: Czech *část obce* (UPU Czech Republic sheet, 02/2018: the part of the municipality is written after the street and house number; Česká pošta, *How to correctly label the consignment*), Slovak *časť obce*, Croatian *naselje* (UPU Croatia sheet, 05/2014). The registers (RÚIAN, DGU Adrese) label it the dependent locality, and the rules now do too: the field is `dependent-locality` (rule `t2-settlement-line`). Postal and Display print the line where the writer put it, and the key reads the words as before (a fixture-level differential over 2,000 dev examples per country shows the canonical form, display form and key unchanged; only the field name differs). A box line, a unit line or a street-type line after the street stays an additional line as written. Slovenia and Hungary are not changed: their registers give no such line. - **Romania: the sector of Bucharest.** UPU Romania sheet: Bucharest is divided into numbered sectors written after the town (`011371 București, Sector 1`). The locality is the town and `dependent-locality` is the sector (rule `t2-romania-sector`), as the Poșta Română register labels them. The printed locality line is unchanged (`011371 BUCUREȘTI, SECTOR 1`) and the key still reads both (`bucuresti sector 1`), so keys of stored addresses do not change. The county stays the admin area. - **Hungary: an addressee in front of the street.** Magyar Posta's addressing notice writes a street name as one to three words before the type (`Kossuth Lajos utca 12.`). With more words, a legal form (`Kft.`, `Zrt.`, `Bt.`) or a person's initial before further words, an addressee was written on the same line with no comma. Where it ends cannot be told from the words, so the rules refuse and the model reads the line (they used to read the addressee as part of the street name). # Last-line orders on web pages The authorities' sheets print the postcode and the locality in one order per country (`docs/address-rules-tier2-*.md`). Web pages write other orders. The rule reader accepts them when the country is known (a hint or a country name in the text), rearranges the lines in memory so the last one reads *postcode, locality* as for every other country, and leaves every span pointing at the words in the input. The rendered `Postal` and `Display` forms are the authority's, as before; nothing is added to the address, and nothing in the text is rewritten. | Form | Example | Countries | Source of the order | |---|---|---|---| | Postcode and locality open the address, the street follows | `00-590 Warszawa, ul. Marszałkowska 10` | every Tier 2 country except Hungary, Romania and Australia, which have their own readers | Poczta Polska, *Zasady poprawnego adresowania* (postcode before locality); the same order is the usual one in the other languages of the postcode-first countries | | Locality first, street, postcode last | `Warszawa, ul. Złota 59, 00-120` | one-line text only | usage; written in lines, a name line above the street is a recipient, so it is not read this way | | Street, locality, postcode apart | `Fő utca 12., Debrecen, 4024` | all | usage | | Locality and postcode at the end of a line, a street elsewhere | `ul. Kościuszki 12, Gdynia 81-001`; `Kraków 31-147, ul. Długa 5` | all | usage | | Country name between the post town and the postcode | `Belfast, UK BT1 1GH`; `Guildford, Surrey, England GU1 3AB` | GB, IE | UPU United Kingdom and Ireland sheets print the country on its own line; written inline it is still the country | | Australian state in full, with the country after | `Melbourne, Victoria 3000, Australia` | AU | Australia Post, *Address Presentation Standards* (the state is an abbreviation or the full name) | | Spanish street known by its name alone | `Gran Vía 25, 28013 Madrid` | ES | the name is the registered street name; there is no type word to read | Limits that keep these readings safe: - The country must be known. In country inference (no hint) a trial read takes these orders only for a postcode form that one country alone writes (`00-590`), because four or five digits at the start of a line are as often a house number. - The city-states (Vatican, San Marino, Monaco) print their own name where the locality would be, so their postcode alone is never preceded by a locality. - The locality is one to four words without digits that do not open a place inside a building and are not a postal-code word (`C.P.`, `PLZ`, `CAP`, ...). - A building, estate or park name above the post town (`Westside Business Park`) is a line of the address, never the post town, so `Galway H91 X2Y3` keeps Galway. - The Scottish council areas `Aberdeen City`, `Dundee City` and `Glasgow City` are counties of the line above the postcode. Not read, left to the model: a voivodeship tail (`31-147 Kraków, woj. małopolskie`), a bilingual locality (`Borriana / Burriana`), dependent-locality lines of the Czech Republic, Croatia and Romania (the real-page gold leaves them unlabelled, so reading them as fields would only add predictions). ## Labels, brackets and island names - **Address labels.** A page may write `Cím: 1053 Budapest, ...`, `Adres: ul. ...` or `Sitz: ...`. The label says what the line is and is no part of the address: it is left out of every field and of the span of the address. A listed label (`Data/T2/t2-address-labels.tsv`: the ordinary words for "address", "registered office", "premises" and "postal address" in Hungarian, Polish, Czech, Slovak, English, German, French, Spanish, Portuguese and Italian, written by us) is dropped at the start of any line when a colon follows it; at the start of the text any single word ending in a colon is dropped. A colon elsewhere stays. - **Brackets.** Round or square brackets around the whole address are dropped (offsets stay those of the original text). - **Island names.** For a hinted Jersey, Guernsey or Isle of Man address: the island's name written right after the town without a comma (`Douglas Isle of Man IM1 1AA`) is the country and the words before it the locality; a town line followed by the island's name and the postcode with no street line (`Ballasalla, Isle of Man, IM9 2AA`) has the town as locality; a trailing `United Kingdom` or `UK` after the island's postcode is kept as the written country text (never rewritten), the island stays the country, and the two names are reported as a `country-conflict` warning. Without the hint the island's postcode still cannot be called Britain's and the text is refused. ## Scoring conventions The scorer's `house_number` for a building and flat number written `12/3` is the whole `12/3` (the gold sets and the reference systems write it so). The library keeps a house number (`12`) and a unit number (`3`); `tools/Eval/Rules` joins the two for scoring when only a slash separates them (set `ONECONTACT_EVAL_NO_SLASH_CONVENTION` to score without the join). On the Polish set the join moves hinted field F1 from 89.96 to 96.39; it is a label convention, not an improvement of the reading. # Addresses whose country cannot be told Many postcodes are written alike in several countries: four digits in Belgium, Austria, Switzerland, Liechtenstein, Luxembourg, Denmark, Norway, Slovenia and Hungary; five in France, Germany, Spain, Italy, Croatia, Mexico, Finland, Estonia, Lithuania and Sweden. When a text has such a postcode, no country hint, and nothing in it that belongs to one country only (country inference, `address-rules-tier2-c-north.md`), the country cannot be told. Before this rule the text was refused with `address.country-undetermined`. A caller reading web pages usually needs the fields all the same: the street, the number, the locality and the postcode are the same whichever country it is. ## Rule When the United States and Canada rules could not read the text and the country-deciding path refused it, the text is read once as each Tier 2 country that can read it (the same trial reads as country inference). Candidates are dropped for the reasons that already veto an inference (a letter outside the language, a Swedish-form postcode written without its space, a province code left in the locality). If at least one candidate is left and **every remaining candidate finds the same fields**, those fields are returned and the country stays unknown. If two candidates differ, the text is refused as before, with one exception (partial agreement): when every remaining candidate reads the same postal code, only the elements all candidates read alike are returned, the country stays unknown, `AddressFacts.FieldsDisputed` (256) is set together with `CountryUndetermined`, and the read carries the warning `address.fields-disputed`. A disputed element (for example a road the candidates split differently) is left out, never guessed. `CountryInferred` is cleared on every undetermined read. Evidence for exactly one country that the inference turned down (a conflict, not an absence) also keeps the refusal. "The same fields" means the same set of (field, offsets) after joining the parts of a road: the house number; the unit parts; the road (street name, type and directionals as one stretch); the locality, admin area, postal code, country name and the lines kept as written. Two countries that split a road into name and type differently still agree on the road. **Language candidates.** A word that the inference tables give to a language (street types and suffixes of the evidence rows, plus the `lang` and `langsuffix` rows; endings under four letters are ignored) removes the countries that do not write that language from the candidates: `Kerkstraat` is Dutch, so a text with a four-digit postcode can only be Belgian (the Netherlands writes two letters after its postcode); `Rue` removes Hungary and Denmark. `language` rows say which languages a country's own rows are written in where that is not all it writes (Belgium's endings are Dutch, Luxembourg's Luxembourgish, Danish and Norwegian understand each other). **Model reads.** A model-only reading of a text the rules refused, with no hint and no known state, ZIP+4 or Canadian postal form, is no longer returned as a United States address: if the Tier 2 candidates agree their fields are returned, otherwise the model's fields come back with the country unknown. An explicit country conflict still refuses. The country-undetermined read happens last: the United States and Canada rules, the Tier 2 rules with a hint or with evidence, and the model read all come first, and a hint always decides. ## What the caller gets | Member | Value | |---|---| | `ParsedAddress.Country` / `PostalAddress.Country` | `Region.Unknown` | | `ParsedAddress.CountryConfidence` | `0` | | `AddressFacts.CountryUndetermined` (new, value 128) | set, in `PostalAddress.Flags` and `ParsedAddress.Facts` | | `ParseDetail.Issues` | a warning with the existing reason `address.country-undetermined` (no new reason code) | | `PostalAddress.Validate()` | `Country` missing (the structural validator already reports a missing country and stops there) | | `PostalAddress.Coverage` | `Structural` (the default for no country) | | `ToString(style, layout, includeCountry)`, `Canonical` | the text as written: the lines of a multi-line input, or the comma-, semicolon- and bar-separated parts of a one-line input, trimmed, joined with a line feed or `, `. Postal and Display are the same; no country line is added. No authority's style applies, so nothing is rewritten | | `Key` | `zz|...` (`address-keys.md`) | | Fields | the first candidate's field names and offsets; each field's standardized text is the text as written. The street parts (`StreetDetail`) are the first candidate's reading of the delivery lines | | JSON | as for any parsed address (the text is stored and read again); the flag survives | With a model package referenced, the model is not asked to decide the country of these texts: it never decides one. # Address expansion dictionaries The expansion of an address (`AddressExpander`) and the words of its key use the dictionary files of libpostal (MIT licence), ingested from pinned data files by `tools/Ingest.LibPostalDictionaries`. Only the files named in `upstream/libpostal-dictionaries/PIN` are used: English and French (United States and Canada) and German, Dutch, Italian and Luxembourgish (the first Tier 2 group: Belgium, the Netherlands, Luxembourg, Germany, Austria, Switzerland, Liechtenstein) street types, directionals, unit and level words, building and post office words, qualifiers, synonyms, street-name words, stopwords, number words, near and cross-street words, entrances and staircases, the French elisions, the German and Dutch concatenated-suffix lists (endings such as `strasse`/`str`, `straat`), and the language-independent ambiguous-expansions list. The pin records the upstream tag (v1.1.4, commit ffafa3fcd687631955453df70906c53d5677ce9a) and the SHA-256 of every file; `tools/fetch/libpostal-dictionaries.sh` fetches them by exact path into `$ONECONTACT_UPSTREAM/libpostal-dictionaries` and checks the hashes, and the ingest tool checks them again before converting anything. Nothing but data files and libpostal's published README was read. ## Upstream format One entry per line, the canonical phrase first, then its variants, separated by `|` (`drive|dr`, `street|st|str`). Lines may lack a final line feed. ## Compiled tables The tool writes one table per file, named `/` (`en/street_types`, `fr/directionals`, `all/ambiguous_expansions`), into the `libpostal-dictionaries` compiled data file (format in architecture.md). A table is UTF-8 text, one entry per line, phrases separated by a tab, in upstream order. Phrases are lower-cased with the invariant culture, runs of white space become one space, empty phrases and repeated phrases within an entry are dropped, and entries left empty are dropped. The same pinned inputs give byte-identical tables. At first use the library reads the tables into a phrase index (a hash of language and phrase to the entries holding it, with the longest phrase length for matching longest first). The packed notice is `notices/libpostal-dictionaries.notice.txt` (MIT licence and copyright line). ## How expansion uses them Text is lower-cased, punctuation becomes a space (a period inside `p.o.` vanishes, a hyphen between digits stays), and the longest phrase at each word is looked up. Street types expand except as the first word of an English address; numbered unit and level words and `number` expand only when a number-like word follows; synonym and street-name entries expand only when the word is not already the canonical form. A word the ambiguous list names, and any single letter, is also kept as written, so `w` gives both `west` and `w`. Ordinals give the digits with and without their ending. The first variant has everything expanded, the last is the input as normalized; the rest are the combinations in between, up to the limit. # Address key, version 1 `PostalAddress.Key`, `GetKey(KeyVersion.V1)` and `AddressKey.Compute` give the key. A published version's output never changes: a rule change ships as a new `KeyVersion` with its own golden fixtures (`testdata/addresses/key-v1-golden.tsv` holds version 1's). The key is built from the parsed parts, not from the text. Segments are joined with `|`: 1. the country, lower case (`us`, `ca`); 2. one segment per delivery line, in order (below); 3. the locality; 4. the state, territory or province code, lower case; 5. the postal code: a ZIP+4 keeps its five digits; a Canadian code loses its space, lower case. Recipient, organization and attention lines never enter the key. **Words** (street names and the locality) are folded: lower case with the invariant culture, diacritics removed from Latin letters by a table of our own (the platform's normalization support is not relied on; `ß` is `ss`, `æ` is `ae`, `œ` is `oe`), punctuation other than a hyphen between digits becomes a space, an ordinal loses its ending (`5th`, `1re`, `2e`, `2ème` become `5`, `1`, `2`), and a word the libpostal street-name and synonym dictionaries know in abbreviated form is replaced by its full word (`st` is `saint`, `mt` is `mount`, `ft` is `fort`). **Delivery lines**: the house number (lower case, hyphens kept) and fraction; a directional as its full word (`n` is `north`, `o` is `ouest`); the street type as the dictionaries' full word (`st` is `street`, `boul` is `boulevard`); units: every numbered unit designator (`apt`, `ste`, `suite`, `unit`, `rm`, `dept`, `#`, `app`, `bureau`, `unité`) reads as `unit`, floors, buildings and the rest keep their dictionary word; a Canadian unit written before the civic number is written after the street as a unit, so `10-123 Main St` and `123 Main Street Apt 10` have one key; a post office box is `pobox n` (Canadian `cp n`) with the station words; routes and military lines keep their designator and numbers (`rr 4 box 87a`, `psc 3022 box 684`). Examples: | Address | Key | |---|---| | 123 North Main Street Apartment 4, Springfield, Virginia 22162-1234 | `us|123 north main street unit 4|springfield|va|22162` | | 10-123 Main St, Montreal QC H3Z 2Y7 | `ca|123 main street unit 10|montreal|qc|h3z2y7` | | 12 Mt Vernon Rd, St Louis MO 63101 | `us|12 mount vernon road|saint louis|mo|63101` | | PSC 3022 Box 684 / FPO AE 09397-0007 | `us|psc 3022 box 684|fpo|ae|09397` | Two addresses that differ only in the unit designator word (`Apt 4` and `Ste 4`) share a key on purpose: the key is for finding duplicates. Different house numbers, different postal codes or different units never share one. ## Addresses without a country An address read with the country left unknown (`AddressFacts.CountryUndetermined`, see `address-country-undetermined.md`) has the key `zz||||`: - `zz` stands where the country code stands; - the street words are the text of the address's `street.*` fields in input order, folded and with the dictionaries' abbreviations replaced as for any other address (the language of the dictionaries is not known, so the default English-and-French set is used); - the locality is folded the same way; the admin area is folded; the postal code keeps only its letters and digits, folded. No country-specific rule applies (no ZIP+4 truncation, no Canadian unit reordering, no particle rules). The same address read with a hint or with a country in the text has a different key by design: `zz|...` and `be|...` are never equal, so an address without a country never matches one with a country by accident. ## Countries described only by their fields An address in a country the rules do not read (the structurally described countries, `Coverage` is `Structural`) has the key `|||||`: the country code in lower case, the street lines folded and joined with spaces, then the other fields in this fixed order whatever the country's own label layout is, each folded as above, the postal code keeping only letters and digits. Before this rule the key had no country segment and followed the layout, so equal fields in two countries shared a key; that was a defect, not a version, because no key version had been published. # Person names Rules only. `PersonName`, `NameParsing.Classify/Split/SplitMany` and `NameHints` read a string, give every part with offsets into the input (UTF-16 chars), and say in `ParseDetail` which issues were met. Nothing is inferred: no gender, origin or salutation, and no part is added that the text did not contain. `CultureInfo` never affects a result; comparison is ordinal and case folding is our own invariant folding. ## Hints `NameHints(Region? country, string? language)` is optional. A hint never overrules the text; it chooses between readings the text leaves open: | Behaviour | Without a hint | With a hint | |---|---|---| | Family name first ("Kovács János") | read given-first | read family-first for `hu` | | Two family names ("Ana García López") | one family name, the rest given/middle | two family names for es, ca, eu, gl, pt | | Lower-case particles after casing restoration | "Van Der Berg" | "van der Berg" for a non-English language (nl, de, fr, es, pt, it, ...) | | Title words that are also names (`don`, `g`, `p`, `hr`, `me`) | not titles | titles in the languages that use them | | Legal-form candidates (companies) | all jurisdictions | the hinted country's first | ## Classes `NameClass` is `Person`, `Company`, `Multiple` or `Unknown`. - **Company**: a legal form of the ELF list (docs/company-names.md), a company word (Ltd-like words, "Holdings", "Associates", ...), a trading-as or care-of marker, or a comma part that is itself a company ("Kvarner Promet d.d., Jansen"). A company word after a given name that is also a surname ("Jane Baker") does not make a company. - **Multiple**: connectors between name parts (`and`, `&`, `et`, `und`, `en`, `e`, `og`, `och`, `ja`, `i`, ...), semicolons and line breaks, and comma lists ending in a connector. "Mr & Mrs John Smith": a title-only part takes the whole name that follows. "John and Jane Smith": a lone word before a full name is a given name and shares the family name. "Maria Garcia y Lopez": `y`/`e`/`i` between two family names is one person (the right side is one word, with particles). Two single words joined by a connector are `Multiple` only when the first is a known given name ("Johnson & Johnson" is `Unknown`). Every part must also read as a person when parsed again alone, otherwise the whole text is `Unknown`. - **Person**: everything else with 2 to 8 words. When nothing proves a person (no title, suffix, nickname, particle, comma order, initial or known given name) the result carries the info issue `person.no-evidence`. - **Unknown**: one bare word, titles and suffixes alone (`person.no-name`), digits or `@`, more than 8 words, more than 8 people, or text that reads as a company once put in reading order (`person.not-a-person`). `PersonName.Parse` still accepts one bare word (flag `SingleWord`). ## Parts Prefix (titles), Given, Middle, Nickname, Particle, Family, SecondFamily, Suffix (generational), ProfessionalSuffix, Order. - Titles are read from the start of each comma segment: academic, military, religious, honorifics, up to three words, dotted runs such as `Prof.Dr.`. A stray dash or ampersand before the titles is skipped. Titles are scoped by language so that words that are names elsewhere are not stripped. - Suffixes are peeled from the end: generational (`Jr`, `Sr`, `II`, `III`), professional (`PhD`, `MD`, `Esq`, ...; a bare one needs a dot or its own comma segment). - Nicknames: quoted (`"Bob"`, curly and guillemet quotes) or parenthesized. A quote mark opens one only at the start of a word and closes it only at the end of one (`O'Brien`, `l'Heureux` are not nicknames); a quoted run that holds a comma is a stray mark and is ignored. - Order: "Family, Given Middle" (trailing particles after the given names move to the family name: "Berg, Jan van den"), given-first, family-first for `hu`. - Particles (`van`, `van der`, `de`, `de la`, `von`, `da`, `dos`, `di`, `le`, ...) stay in `Particle`, apart from `Family`. - Spanish-style second family names with a hint; a three-word name whose middle word is the second half of a compound given name ("Ana Maria Garcia") keeps one family name. ## Canonical form Always given-first reading order: "Smith, John" gives `John Smith`, and Hungarian "Kovács János" gives `János Kovács`; `Order` remembers the input order. This keeps the JSON round trip stable. Nicknames are written `"Nick"`, professional suffixes after a comma (`, PhD`), generational suffixes inline. Mixed-case input is kept as written. A reparse of a family-first canonical form with the `hu` hint is not idempotent, by design. ## Casing restoration Applied only when the whole input is one case (at least two letters, all upper or all lower). Each restored part says so with the info issue `person.casing-restored`, and keeps the offsets of the original. - Capital first letter of each word; hyphen and apostrophe parts too (`O'BRIEN` gives `O'Brien`, `ANNE-MARIE` gives `Anne-Marie`). - `Mc` followed by a capital (`MCDONALD` gives `McDonald`). `Mac` is not restored: `MACDONALD` gives `Macdonald`, because `Mac` prefixes are too often not a prefix (`MACHADO`). - Particles are lower case only with a non-English hint (not Irish): `VAN DER BERG` gives `Van Der Berg` without a hint and `van der Berg` with `nl`. ## Person key V1 See docs/name-keys.md. ## Black-box comparison `tools/Oracles/Names` runs, from released packages and never reading them: python-nameparser 2.8.2 (used through `HumanName`, as its documentation describes), probablepeople 0.5.6, cleanco 2.3 (Python, in a `uv` venv) and NameParserSharp 1.5.0 (NuGet, in the oracle project only), over `testdata/names/gold.tsv`: 522 synthetic person and company strings with gold parts, 28 languages, made by `make-gold.py` with seed 20261105. `score.py` scores token accuracy (each gold part's words get the reference's label), class accuracy and legal-form detection. The gold set is made from the same kind of templates our rules were written against, so our scores are near 100% by construction. The comparison is information about the references' coverage of these languages, not a quality claim for real-world text. ## Iberian, Irish, Maltese and Slavic forms Rules only; they read the words of a name and say nothing about gender, origin or salutation. - Titles `Mons.`, `Msgr.`, `Monsignor`, `Mestre`/`Mestra`, `Revd`, `Rt Hon` are prefixes (the ambiguous ones only with a language hint). `Sen.`, `Senator`, `Hon.`, `Rev.` and `Fr.` were already prefixes. - Portuguese: a particle two words from the end keeps its family name: `Ana Maria dos Santos Ferreira` is given `Ana`, middle `Maria`, particle `dos`, family `Santos`, second family `Ferreira`. In a `Family, Given` text a connector between two family names (`Garcia y Lopez, Maria`) is one person. - Irish `Mac` standing apart (`Padraig Mac Giolla`) is a particle with an Irish hint; `Ó`, `Ní`, `Nic`, `Uí` and `Mhic` already were. Maltese `ta`, `tal`, `tas`, `tat` and `il` written apart are particles with a Maltese hint (`Josef ta' Cassar`); written with a hyphen (`Il-Borg`) they stay in the family name. - Czech, Slovak, Croatian and Slovenian: two words read as `Surname Given` when the second is a listed given name and the first is not, or the first is a feminine `-ová` surname form and the second is not (`Novák Jan`, `Nováková Jana`). Names that are also common surnames (Marek, Martin, Matej) are not listed, so `Marek Jan` and `Jan Marek` keep the order as written. The given-name list is our own. - Degrees: `MgA.`, `BcA.`, `Prom.`, `PharmDr.`, `ThLic.`, `ThMgr.` (cs, sk), `dipl. iur.`, `dipl. oec.`, `mag. oec.`, `dr. med.`, `dent.`, `bacc.`, `spec.`, `struč.` (hr) are prefixes; `CSc.`, `DrSc.` and `DiS.` after a comma or with dots are suffixes. - A title after the name is a title: `Novák Jan Ing.`, `Jana Nováková Ing.`, `Ivan Horvat dipl. ing.` give the prefix `Ing.` (`dipl. ing.`) when the language hint lists the words as titles, they end in a period, have at least two letters and a name stands before them (a single letter with a period, `P.`, is an initial). With a title in front already, the later one joins the professional suffixes. Registers and signatures of the Czech Republic, Slovakia and Croatia write degrees this way; the words are the lists already used for the leading position. - More credentials after a comma or a name are professional suffixes: `EMBA`, `BBA`, `MIM`, `ACCA`, `FCCA`, `FACCA`, `FRCP`, `MRCP`, `MBChB`, `MPhil`, `MEng`, `FCA`, `CMA`, `ACMA`, `FCMA`, `CGMA`, `FCPA`, `MHA`, `FRM`, `PRM` are suffixes anywhere; `CIA`, `CFE`, `MArch`, `MSt`, `MRes`, `MAS`, `ACA`, `CAIA` need a dot or a comma (they are also words). The casing of each is fixed when casing is restored. - The given-name lists used for the Czech, Slovak, Croatian and Slovenian `Surname Given` order, and for telling two people from a company, were widened with common names of those languages and the Nordic ones (our own lists; names that are also common surnames are still left out of the Slavic list). # Person key `PersonKey.Compute(PersonName, KeyVersion)` and `PersonName.Key` give the aggressive, versioned comparison key. Canonical forms stay faithful; only the key loses information. A published version's output never changes: a rule change ships as a new `KeyVersion` with its own golden fixtures (`NameRuleTests`). ## V1 For a name, in this order, joined by single spaces: 1. **Given**: the given name folded (below), then replaced by its formal name when it is a listed nickname ("bob" gives "robert", "liz" gives "elizabeth"). 2. **Middle**: the first letter of each folded middle word. 3. **Family**: particle, family name and second family name folded and concatenated without spaces ("van der Berg" gives `vanderberg`). Dropped: titles, generational and professional suffixes, the nickname part, the family connector (`y`). **Folding**: invariant lower case, then a fixed table maps Latin letters with marks to their base letter (`é` to `e`, `ø` to `o`, `ł` to `l`, `đ` to `d`, ...), `ß` to `ss`, `æ` to `ae`, `œ` to `oe`, `þ` to `th`, `ð` to `d`; combining marks (U+0300 to U+036F) vanish, so decomposed text folds like precomposed text; every other character that is not a letter or digit (apostrophes, hyphens, dots) is removed. No Unicode normalization tables are involved. **Nicknames**: our own list (`NameLexicon`), written for this repository; no external list is used. Only unambiguous pairs are in it. Short forms that are also names or words of their own (Al, Sam, Chris, Pat, ...) are left out, and so are equivalences across languages (José is not Joseph). Never inferred: gender, origin, language. Golden examples (fixtures in `NameRuleTests`): `Bob Smith` and `Dr. Robert "Bobby" Smith Jr., PhD` both give `robert smith`; `Robert James Smith` gives `robert j smith`; `Liz O'Brien` gives `elizabeth obrien`; `Anna van der Berg` gives `anna vanderberg`; `José García López` with the hint `ES` gives `jose garcialopez`. # Company names `CompanyName` and `CompanyNameParser` read a company name into: leading article ("The"), base name, legal form (with its ISO 20275 ELF code and a canonical abbreviation), "trading as" part and "c/o" part. Every part carries offsets into the input. Rules only; the legal form comes from the compiled GLEIF list, so a new jurisdiction is a data update. Nothing is invented: a form that fits several codes has no code (see Candidates), and the base name is never completed or corrected. ## The ELF data set Source: the GLEIF Entity Legal Forms code list (ISO 20275), version 1.6 of 2026-02-19, CSV, pinned in `upstream/gleif-elf/PIN` with its SHA-256 (4002 rows, 3599 codes, 130 country codes). Licence: CC0, as GLEIF states on its open-data and LEI-data terms pages; the credit line is in `THIRD-PARTY-NOTICES.md` and in the notice packed with the library. `tools/fetch/gleif-elf.sh` fetches the file by exact URL and checks the hash; the ingest tool `tools/Ingest.GleifElf` checks the CSV header names, then writes the tables below at build time. The same pinned file gives byte-identical tables. Table `forms`: a 32-bit little-endian row count, then per row one byte `active` (1 for ACTV, 0 for INAC) and eight strings, each a 16-bit little-endian byte length followed by UTF-8: code, country code, sub-division code, local name, language code, transliterated name, local abbreviations, transliterated abbreviations (abbreviations separated by `;`). Rows are sorted by country, code, language, then file order. Table `version` holds the list version text. ## Matching A legal form is looked up in an index of the folded text (see docs/name-keys.md for folding) of every name and abbreviation, plus our own alias list for spellings people use every day that the list does not carry (CZ/SK `s.r.o.`, `a.s.`; PT `Lda`; IE `DAC`; FR `SARL`; ES `S.A.`, ...). Each alias names a listed form of its country; one that no longer names a listed form is skipped, never fails the load. The aliases are ours, written from the forms' published names. - The form is taken from the end of the name, longest match first. A prefix form is taken only if it is an abbreviation of the hinted country, or one of a few unambiguous prefix abbreviations (`UAB`, `SIA`, `ZAO`, ...). - Forms of at most two letters need capitals or real dotted letters (`S.A.`, `A/S`), or a country hint, so that a word such as "as" or "sa" is not a form. - `d/b/a` and `DBA` are trading-as markers, never legal forms. - A hinted country restricts the candidates to its rows when it has any; if it has none, the result says the form lies outside the hinted country (`company.legal-form-outside-country`). - Inactive (INAC) rows are used only if no active row matches (`company.legal-form-inactive`). ## Candidates and the ELF code The result lists every distinct code the written form can mean (`Candidates`: code, country, name, active). The ELF code is set only when exactly one code remains; otherwise it is empty and the info issue `company.legal-form-ambiguous` is reported. ## Canonical abbreviation For a single code the abbreviation is taken from the rows that carry the written form: the one with the fewest letters; among equals mixed case over capitals when it is longer than three letters ("Pty Ltd" over "PTY LTD"), then fewer punctuation marks, then a capital initial. The canonical text replaces what was written ("Limited" gives "Ltd", "Aktiengesellschaft" gives "AG"), except when the two differ only in case ("Oy" is not rewritten to "oy"). A composite form ("GmbH & Co. KG") is kept as typed. With several codes the common abbreviation is used if there is one. ## Other parts - **Leading "The"**: split off when more words follow. It is not part of the base or the key. - **Trading as**: `d/b/a`, `dba`, `doing business as`, `t/a`, `trading as`, `trading under`; the canonical text writes `d/b/a` or `t/a`. An empty trading name is `company.empty-trading-name`. - **Care of**: `c/o`, `care of`. A text that starts with `c/o` and holds nothing before it is read as the company after it (`company.care-of-only`). - A comma between words is kept ("Smith, Jones & Partners"), a comma before the legal form is not. ## Company key V1 `base | token`: the base name folded to lower-case words (an ampersand is the word "and"; the leading "The", the trading name and the care-of part are left out), a bar, then the token of the legal form: - the ELF code when the written form maps to one code; - otherwise `~` followed by the folded common canonical abbreviation, with English synonyms unified ("Incorporated" and "Inc" give `~inc`, "Limited" and "Ltd" give `~ltd`), computed without any hint so that the key never depends on one; - nothing when there is no form. Examples (fixtures in `NameRuleTests`): `The ACME Muster GmbH & Co. KG` gives `acme muster|8Z6G`; "Acme Holdings Ltd" and "ACME HOLDINGS LIMITED" share a key; "Acme" and "Acme Ltd" do not. A published version's output never changes; a rule change ships as a new `KeyVersion`. # Extraction `RecursiveAI.OneContact.Extraction` reads the contact information a page states in structured markup, and the contact details in its visible text, and returns them as person and organization records. The first half of this document is what the markup readers do; the visible-text half starts at "Visible text". ```csharp IReadOnlyList records = ContactExtractor.Extract(html); // string or bytes var extractor = new ContactExtractor { Sources = ExtractionSources.Markup, DefaultRegion = Region.US }; ExtractResult result = extractor.ExtractHtml(bytes, contentType: "text/html; charset=utf-8"); foreach (ContactRecord r in result.Records) { /* r.Name, r.Phones, r.Emails, r.Websites, r.Addresses, r.Links */ } ``` Nothing is fetched and nothing is stored. **Personal data:** names, emails, phones and addresses are personal data. Extraction runs offline on exactly what the caller passes in, and the library keeps no copy; responsibility for a lawful basis and for compliance with data-protection law (for example the GDPR) for what is extracted and what is done with it sits with the caller. ## Result model | Type | What | |---|---| | `ExtractResult` | `Records`, every `Fields` flat in document order, `Ungrouped` fields, `Issues`, the `Sources` that ran and the `OffsetUnit` | | `ContactRecord` | `Kind` (`Person`, `Organization`), `Fields`, accessors per `FieldKind`, `Links`, `Disagreements`, `Span`, `Confidence` | | `ExtractedField` | `Kind`, `Source`, `Raw`, `Text`, `Span`, `Confidence`, `Flags`, `Normalized` (a `ContactValue`), `Issue`, `Facts`, `Label`, `RecordId`, `BlockId`, `Duplicates` | | `FieldKind` | `Name`, `Organization` (the employer by name), `Phone`, `Email`, `Website`, `Address`. New kinds may be added; consumers keep a default branch | | `FieldSource` | `JsonLd`, `Microdata`, `Rdfa`, `HCard`, `Link`, `AddressElement`, `Text` | | `RecordLink` | `WorksFor`, `HasMember`, `ParentOrganization`, `SubOrganization`; always stored in both directions | **Raw and text.** `Raw` is the source slice at `Span`, exactly as written (character references and JSON escapes undecoded; newlines as HTML parsing normalizes them, CR LF and CR to LF). `Text` is what the normalizer was given: references and escapes decoded, markup removed, white space collapsed, address lines separated by `\n`. Where a value is put together from several pieces of markup (a given and a family name; address parts) the field carries `Composed`, `Raw` spans all the pieces and `Text` is the joined value. **Offsets.** `Span` counts bytes of the original input when the input was bytes (or a `Document` OneHtml parsed from bytes), through OneHtml's offset map, so it is right for UTF-8 with multibyte text, other encodings and a byte order mark. For a string it counts UTF-16 code units of the string as given, carriage returns included. A `Document` parsed from a string has already had its newlines normalized, and its offsets are into that normalized text. `HasSpan` is false only for text the parser had to rewrite. **Confidence.** Markup that names the property outright counts 1.0; a `mailto:`/`tel:` link counts 0.9 (a link only suggests the role); an `address` element 0.7; a value that could not be normalized counts half. A record's confidence is its best name's, or 0.5 without a name. No model is involved at this stage. **Normalization.** Each value goes through the core type of its kind (`PersonName` or `CompanyName` for names by the record's kind, `PhoneNumber`, `EmailAddress`, `WebAddress`, `PostalAddress`). A value the core refuses stays as text with `Unnormalized` and the core's reason in `Issue`. `Facts` lists the fact flags of the normalized value by name. - **Phones** need a region. An international number (`+`) needs none. Otherwise the region is the country of an address in the same record, else the region of the page's `lang` (`en-US` gives US; `en` gives none), else `ContactExtractor.DefaultRegion`. When one is used the field carries `RegionInferred`; when none exists the field stays unparsed with `phone.region-required`. A region is never guessed. - **Free-text addresses** (a string `address`, an `address` element) are parsed in the same fallback region, and stay unparsed with `address.country-required` without one. Core parses free text only for the countries it has rules for. - **Structured addresses** (schema.org `PostalAddress`, address parts of an h-card) are not parsed: the parts are taken as given. The country is the page's `addressCountry` (a code, a name the core country data knows, or a few common aliases such as USA and UK). Without one the address keeps no country and carries `address.country-required` as its issue. - **Emails** are the `mailto:` address before any `?` header fields, percent-decoded (RFC 6068 §2); a link with several recipients gives one field each. **Phones from `tel:`** are percent-decoded with their parameters (RFC 3966 §3). ## Readers Every reader reads hidden markup too: structured markup is meant for machines. A field inside markup hidden by a `hidden` attribute, `aria-hidden="true"`, or an inline `display:none` / `visibility:hidden` on it or an ancestor carries `Hidden`. Style sheets are not consulted. - **JSON-LD.** `script` elements whose `type` is `application/ld+json` (media-type parameters ignored). A strict RFC 8259 reader keeps the offset of every value; a block with a syntax error (trailing comma, comment, raw control character, text after the value, nesting deeper than 64) is rejected whole with an `ExtractIssue` of kind `MalformedJsonLd`, and the other blocks and readers still run. Understood: `@type` (string or array; `schema:Person`, `http://schema.org/Person` and `Person` are the same), `@id` references (resolved across blocks, cycles end), `@graph`, top-level arrays, `@value`, `@list`, `@set`. Contexts are not expanded: a context that renames terms is not honoured. - **Microdata.** Top-level items are elements with `itemscope` and no `itemprop`. Property values follow the HTML standard's rules (`meta` content, `href` of links, `src` of media, `data`/`meter` value, `time` datetime, otherwise the text). `itemref` is not followed. - **RDFa Lite.** `vocab`, `typeof`, `property`, `resource`, `prefix`, for schema.org only (the nearest `vocab` names schema.org, or the type is written `schema:Name` or as a schema.org address). Values: `content`, then `href`/`src`/`datetime`, then `resource`, then text. - **h-card.** microformats2 (`h-card`, `p-`/`u-` property classes) and classic hCard (`vcard` with `fn`, `org`, `adr`, `tel`, `email`, `url`, `street-address`, `locality`, `region`, `postal-code`, `country-name`), with the implied name and url of a link card, `abbr[title]`, and the `value` child of a `tel`. A card nested as another card's `p-org` is that card's organization and no card of its own. A card is an organization when it names itself as its own `org` (`fn org`) or its name carries a legal form; otherwise a person. - **Links.** `a`/`area` with `mailto:` or `tel:` (scheme case-insensitive; entity-encoded schemes arrive decoded from the HTML parser). `Label` holds the link text when it differs from the value. A link inside a structured item joins that item's record; one that a structured reader already took as a property is not read twice. - **Address elements.** `address` elements that are not a structured property. The contact links inside are read as links; the rest of the text is the address. `BlockId` (the element's index in the document) lets later grouping tie the loose fields of one block together. **Types.** Organizations are schema.org's `Organization` hierarchy as schema.org lists it (`LocalBusiness`, `Store`, `Restaurant`, `Corporation`, ...). A type outside it is not read as an organization, and a non-contact item (`Article`, `Product`) is searched for records inside it. ## Grouping and de-duplication Records follow the markup's nesting. Organization `employee`/`member` give person records linked `HasMember`/`WorksFor`; a person's `worksFor` (object form) gives an organization record, and always an `Organization` field with the name at the name's own position (text form gives the field only); `parentOrganization`, `subOrganization`, `department` give `ParentOrganization`/`SubOrganization` links. A `contactPoint` or `location` adds its phone, email, url and address to the record that holds it (`Label` carries the `contactType`). Direct address parts on a record form its address. Records come in the order their markup begins. A record is merged into an earlier one of the same kind when they share a name key, or a contact key: an email address that is not a role account, or (organizations) a website that is not a social profile. Phone numbers never decide: a main line is shared by the people behind it. Merged records keep every field. Within a record a value read twice (same kind, same normalized key, or the same text when unnormalized) is one field; the other sightings are in `Duplicates`, each with its own source and span. ## Disagreements Nothing is silently picked. `ContactRecord.Disagreements` lists, per kind, fields that do not fit together: two different names, or phone/email/website values from different sources whose sets share nothing (a stale JSON-LD number against the one in the visible link). All the values stay in `Fields`, flagged `Disputed`. Addresses and employers are not compared: two spellings of one address have different keys, and a person can have several employers. ## Limits JSON nesting 64; microdata, RDFa and h-card item nesting 32; 200,000 item visits and 100,000 fields per page, after which the result holds what was read and an `ExtractIssue` of kind `LimitExceeded` says so. The input limits are `ContactExtractor.Limits` (`ExtractionLimits`: `MaxInputLength` 64 MiB, `MaxNesting` 4096, `MaxNodes` 4,194,304, `MaxFields` 100,000); an input past the first three throws `ContactExtractionException` (with `LimitName`, `Limit`, `Actual`, and the HTML parser's exception as `InnerException`), and past `MaxFields` the result holds what was read. The readers use explicit stacks, so deeply nested HTML does not exhaust the call stack. A core parser that throws on a hostile value does not cost the page: the field is kept unnormalized. ## Visible text `ContactExtractor.Sources` defaults to `Markup | Text`. The text half reads what a reader would see with rules, and with a model package referenced it adds what the rules left (below). **What is read.** Text nodes of the OneHtml document, in order, split into lines at block-level elements, `
`, list items and table cells. Each run keeps its landmark (`header`, `footer`/`contentinfo`, `main`, and so on, from the accessibility tree) and its nearest heading, and every character maps back to a byte offset in the original page (a UTF-16 offset when the page was given as a string). Non-text elements (`script`, `style`, `template`, `noscript`, `iframe`, `svg`, `head`, form lists and the like) are never read. **Cascade.** A node is skipped when its computed `display` is `none`, its `visibility` is `hidden` or `collapse`, it has the `hidden` attribute, a zero font size (also written without a unit), or an ancestor has `aria-hidden="true"` (that prunes the subtree). `visibility` is inherited, so a child that sets `visibility: visible` inside a hidden parent is read. Structured markup is read by the markup readers regardless of visibility, as it is meant for machines. **Plain text.** `ExtractText` takes a CRM note or comment. Lines are the unit; a blank line or a line of dashes, underscores or equals signs starts a paragraph. Offsets are UTF-16 offsets of the string as given, line ends included. **Candidates.** Emails (our parser), websites (our parser and the PSL; a bare domain needs an ICANN suffix and must not look like a file name), phones (the core matcher at `Valid` leniency when the page or caller region is known, plus the shape rules for labelled numbers and `+` numbers), US/CA addresses (the rule parser over block-bounded windows) and names and organizations by the rules in "People and organizations by rules" below (a `Name:` or `Contact person:` label still makes a name certain). The region is never guessed: an address without a region stays unnormalized with `address.country-required`, and an unlabelled national phone without a region is dropped. **Confidence** is fixed per rule: emails 0.9 (0.8 when decoded), phones 0.65–0.85, websites 0.6–0.85, addresses 0.7, labelled names 0.8, Cloudflare emails 0.9. ## The span model (from 1.1) Version 1.0 registers no span model, so extraction is rules and markup; this section describes what the 1.1 model packages add through the `ContactModels` seam. **Plain text only.** The Tier 1 package registers its contact-span head, and `ExtractText` uses it. A web page (`Extract` over HTML) does not: scored on real pages, the head made the record and address rows worse than the rules alone, and no model may make a row worse than rules. Pages are read by the markup readers and the rules, with or without the package. When a contact-span model is registered (`ContactModels.Spans`, `docs/models.md`), the text half also runs it over the visible text, in pieces of up to 40 lines or 3,000 characters. Rules first: the model's spans are considered only after every rule candidate has been taken, and a span that touches any character a rule took is dropped, so a rule's field is never replaced, shortened or split. - A model span becomes a field of the same kinds the rules make: `person` a name, `company` an organization (the grouping decides whether it names the record or its employer, as for a `Company:` label), `address`, `phone`, `email`, `website`. It must have a mean probability of at least 0.4. - The core parsers still decide: a model email or website that does not parse (a website needs a registrable domain) is not a field, and a model phone is kept only when the numbering plan accepts it (and a region is known). A model address is parsed by the rules and then the address model like any other free-text address; nothing is invented. - Such a field has `Decision` = `DecisionSource.Model` and `Confidence` = the model's (halved when the value could not be normalized, as for every field); every other field says `Rule`. Its `Source` is `Text`. - **Blocks.** The model also says where a contact block begins. Grouping treats a block start as a hard edge: fields on or after the start line are not put in a group with fields before it. The layout rules (containers, landmarks, headings, four lines, a second email or address) still apply as before, so the model can only split a group, never join two. - **Without a registered span model** (the default) extraction is rules and markup. Nothing is downloaded. ## People and organizations by rules The rules find a person or an organization in visible text, with or without a model, and with the same offsets and confidence as every other field. A name is only taken when something beside it says what it is; capitalized words alone are never enough. Job titles are evidence only and are never returned. - **Person.** Evidence is an email address, phone number or address in the neighbouring lines; a job title beside it; a cue ("contact", "ask for", "signed", "from"); a closing line above ("Best regards,"); a name alone above a job line or a contact line; a single first name after a greeting ("Hi Lucile,"). The span includes an honorific in front and an ending such as `Jr.`, `Esq` or `CPA` behind (the convention of the development labels). Words of web chrome ("Customer Service", "Privacy Policy"), places, region codes and stop words are refused as names. - **Organization.** A legal form the core company parser accepts (`Acme Widgets, Inc.`), extended over `dba` and `c/o` parts and cut at a connector word in running prose; the name after a copyright sign ("© 2026 Harbor Freight LLC."); an employer after a job title and "at"/"for"; a cue word plus a business word; the company line of a signature. A leading "The" is kept. - **Side by side.** Two organizations in sibling inline elements do not merge into one name. - **Markup agrees.** A text name equal to a name that a markup record already holds takes that record's kind, so "Harbor Freight" in a page's text is an organization when its JSON-LD says so. - **Email.** An address that ends in a file extension (`logo@2x.png`) is not an email unless the ending is a real top-level domain. ### People in typed text Text a person typed (a string, a note, a comment) is read with more rules than a web page, because the same shapes mean something else on a page: a byline, a testimonial, a product name. `ExtractText` and text items turn them on; a web page keeps only the rules above plus the ones marked "any input" here. Each needs a reason in the text itself; none builds a name from an email address or a domain. - **A given name and a family name** (two or three words, the first a given name from the lists). Given names from the core parser's list and the 2,200 commonest US given names; family names from the 5,000 commonest (US Census 2010 and SSA lists, public domain, rendered into `NameListData` by `tools/data/names/extraction-lists.py`). Given names that are also everyday words ("Will", "Rose", "Grace") never start a name on their own. A sentence opener in front ("Yesterday Maria Lopez") is left out of the span. - **A cue and a name the lists know.** A small word that takes a person as object ("to", "with", "by", "for", "from", "per", "via", "told", "met") counts when the first word is a given name or the last a family name. Stronger cues ("spoke with", "ask for", "attendees:", "referred by") count for any name. - **A verb after the name.** Verbs of answering and writing ("replied", "wrote", "texted", "called") take a person as subject for any two or three capitalized words; other verbs ("said", "was", "will") need a name from the lists. - **A list.** A name next to a name already found, with only commas, semicolons, "and" or "&" between them, or on the line below a name that stands alone on its line. - **Lower case.** A given name and a family name in lower case count with a cue in front, a contact field beside them, a speech verb after them, or an email address that spells them. - **Capitals.** The same rules apply to capitalized text. A label word in front ("CUSTOMER", "CLIENT") and a reference word or verb behind ("RE", "CALLED") are not part of the name. - **Last, First.** `Lopez, Maria` at the start of a line or after a cue; it needs a given name second or a contact, a job title or a header label after it, and an opener ("Sure, Jim"), a place or a business word before the comma refuses it. - **A first name alone** after a cue ("spoke with Dan", "Thank you, Dan.") or in front of a verb ("Sofia called"), for a common given name; lower case only after the strongest cues. - **Plain text only:** a title in front ("Dr. Patel", "Mrs. Alvarez"; "Dr" without a dot after a capitalized word is a street; on a page a title alone is a byline or a testimonial), a leading label word ("CUSTOMER Shane Wilfley"), a verb or reference word at the back ("Elian Messer called"), the greetings "thanks", "good morning" and "thank you", a header label and colon ("From:", "Attendees:", "Ref:"), and an email address in the same or a neighbouring line whose local part spells a word of the name. On a page the same shapes are menus, bylines and addresses of a business, and they added predictions that matched no contact. - **Never a contact:** a byline ("By Dana Reyes | May 3"), an attribution after a dash, and a name on its own line under a quotation. Development rows for these rules are in the repository's `tools/Eval/extraction/crm-texts.py` (synthetic notes and comments from our own templates, with negatives that look like names: "Will Call", "Rose Hill", "Lincoln, Nebraska"). Word lists (titles, cues, places, region codes, industry words, legal-form words) are our own, in `NameVocabulary` and `NameLists`. ## The span model on real pages (from 1.1) On a web page the span model is not used: scored on real pages, it lowered the record and address rows below the rules and markup alone, and no model may make a row worse than the rules. Pages are read by markup first, then by the rules over visible text. A gated way of letting the model add people and organizations near fields the rules found was also scored and left off. The measured rows are kept in `docs/maintainers/extraction-measurements.md`. ## Obfuscation Each decoded field carries `Deobfuscated`; its offsets span the text as written (the obfuscated form). - Token substitution: `[at]`, `(at)`, `{at}`, `-at-`, ` at `, `AT`, and `[dot]`, `(dot)`, ` dot ` for the dot. A bare ` at ` counts only when a dot word also appears, so "meet me at example.com" is not an email. Decoded addresses need a real ICANN suffix. - Entity-encoded letters, digits, `@` and `.`. - Cloudflare `data-cfemail`: hex, the first byte is the XOR key for the rest. - Reversed text: `direction: rtl` with `unicode-bidi: bidi-override` is read in display order (UAX #9). - A value split by inline markup (`john@example.com`), hidden junk or zero-width characters. ## Text grouping Fields group into a record when they share a qualifying container (`tr`, `article` and `address` always; other containers only when they hold two kinds of field), the same landmark and heading, and sit at most four lines apart. A second different email or address starts a new group. A group needs two kinds of field. A group with no certain name has kind `Unknown`; a `Name:` label makes a person, and a lone `Company:` label an organization. Unknown groups merge into a markup record that shares an email (role accounts count for organizations only), a website or an address (organizations only); a lone text field joins the single record that already holds its key. Disagreeing values stay, flagged `Disputed`. ## Pages of one site The fourth input is a set of pages the caller supplies from one site. Nothing is fetched: the extractor reads what it is given, and the id of a page is only a label (normally its URL) that must be unique in the set. ```csharp IReadOnlyList records = ContactExtractor.Extract(new[] { ContactPage.FromHtml("https://example.com/", homeHtml), ContactPage.FromHtml("https://example.com/contact", contactBytes, contentType: "text/html"), ContactPage.FromText("note-17", crmNote), }); SiteExtractResult site = new ContactExtractor { DefaultRegion = Region.US }.ExtractSite(pages); SiteExtractResult streamed = await extractor.ExtractSiteAsync(pageStream); // IAsyncEnumerable SiteExtraction run = extractor.BeginSite(); run.Add(page); SiteExtractResult done = run.Complete(); // the engine form ``` **Each page is read on its own**, with the readers and region rules above, so a field's `Span` is in its page and in that page's `SitePage.OffsetUnit` (bytes for a page given as bytes, UTF-16 code units for a string or a text item). `ExtractedField.PageId` names the page. Pages are parsed and released one at a time; what stays is the values read (strings and spans), plus the input of a page that needs the site's region (below), held by reference until `Complete`. **Records merge across pages** on the keys a single page uses: a name key, or a non-role email, or (organizations) a non-social website; phones never decide. Text groups of kind `Unknown` join as they do on a page (a shared email, an organization's website or address, a role account an organization lists). One addition applies to a page too: two organizations whose addresses share none are branches, and a shared website or email alone does not merge them (a shared name does). The result lists records in the order of their first page, then their markup; `ContactRecord.Pages` lists the pages that contributed and `Span` is the position on the first of them. **Fields.** A value seen more than once for a record is one field; the other sightings are in `Duplicates`, each with its own page, source and span. Across pages the sighting in markup is the primary one and the same text found in the visible text of another page is its echo. A value found in text that no record claims joins the one record that holds it, from any page (a `mailto:` link, which only suggests a role, does not). Fields come in page order, then document order, and `Id` counts across the site. **Disagreements** are kept, never picked. They are computed per (page, reader): a record's phones, emails or websites disagree when two of those groups each report values and share none (a stale number on one page against the one on another), and two different names are always a disagreement. All the values stay, flagged `Disputed`. Two different values on different pages are visible this way even when each page is consistent. **The site's organization.** `SiteExtractResult.Organization` is the organization record found on the most pages (the first on a tie), which is what footer, contact and about pages describe together; markup and text sightings merge into it by the rules above. Other organizations (branches, partners) are records of their own. **Persons are linked** (`WorksFor` / `HasMember`) only where the pages say so: through the markup or the same text block, as on a page, or when the person names an employer that is a record's name on any page. Sharing a page with an organization links nothing (a quoted customer is not staff); `Pages` still shows where a person came from. **Region.** `SiteExtractResult.Region` is the one region the site's own content points to: the `lang` regions of its pages and the countries its markup gives for addresses, when they all agree. A page that has values needing a region (a national phone number, a free-text address) and names none is read a second time in that region; its values carry `RegionInferred` and its `SitePage.RegionFromSite` is true. A site with two regions has none; a region is never guessed, and a `DefaultRegion` the caller gave always stands in place of it. **Errors and limits.** A default `ContactPage`, a duplicate id or a null set throws `ArgumentException` / `ArgumentNullException`; a page past the extractor's limits throws `ContactExtractionException` as for a single page. Problems of a page (`MalformedJsonLd`, `LimitExceeded`) come back in `Issues` with the page id. The single-page limits apply per page; the number of pages is the caller's to bound. # Models > **From 1.1.** The model packages described here ship in 1.1, after OneNLP 1.0. Version 1.0 runs on rules and markup alone; the seam in core that registers a model (`ContactModels`) is part of 1.0. The Tier 1 + Tier 2 package `RecursiveAI.OneContact.Models` is described in `models-superset.md`. Normative notes on how trained models reach the parsers, what a model package carries, and how a model is trained. Rules first: a model only fills what the rules leave open. ## What a model does and does not do - **Addresses.** Rules read the last line (postal code, state or province, city), the delivery line and the country (`docs/address-rules-us-ca.md`). Input the rules refuse with `address.model-required`, `address.no-last-line` or `address.no-delivery-line` goes to the address model, which labels the words of the whole text; the parser turns the labels into the same `ParsedAddress` the rules make. When the rules succeed, every field they found stays; the model only labels the lines the rules keep as written (recipient, organization) where it can. A model field says `DecisionSource.Model` in `ParseDetail` with the model's confidence; a rule field says `Rule` with confidence 1. The standard form of a model field uses the authority tables where they have an entry (state, province, street type, directional, unit designator) and the upper-cased words otherwise. Nothing is invented: a field the model did not label is absent, and the country must come from a country name, the state or province, the postal code or the hint. - **Names.** Rules classify a string as person, company, several people or unknown (`docs/names.md`). A string the rules leave `Unknown` goes to the name model. Its class counts when more than half of the words (weighted by their probability) vote for it. A person needs a given or a family name from the model's parts; a company is read by the company rules with the class decided by the model; several people are reported as a class only, because the model labels words and does not split people. A class or part a rule found is never replaced. The rules also take two or three plain words as a person for want of a sign of a company (their issue `person.no-evidence`); that is a default, not a rule hit, so when the model says company or unknown with confidence 0.6 or more it decides instead. - **A state or province from the model must be one.** Of several admin-area spans the last one the authority tables know is taken, and a rejected one right before the city becomes the start of the city (`new york ny`). - **Without a model package** the refusal stays: `address.model-required`, whose description names `RecursiveAI.OneContact.Models.Tier1`. Nothing is downloaded, ever. ## The seam in core `RecursiveAI.OneContact.Models` (core, BCL only): | Type | Role | |---|---| | `IAddressFieldModel.Tag(string)` | Labelled spans (`street.name`, `locality`, `recipient`, ...) with offsets into the text and a confidence. | | `INameModel.Analyze(string)` | A `NameClass`, its confidence and labelled parts (`Given`, `Family`, `Base`, `LegalForm`, ...). | | `IContactSpanModel.Find(string)` | The six contact fields (`person`, `company`, `address`, `phone`, `email`, `website`) as labelled spans, and the offsets where contact blocks begin. | | `ContactModels.Register(packageId, rank, address, name, spans)` | A package makes its models known (`spans` is optional). The highest rank wins: Tier 1 is 1, the Tier 1+2 superset 2, the accurate tier 3. | | `ContactModels.Address`, `.Name`, `.Spans`, `.PackageId` | What the parsers use; `null` when no package is referenced. | Registration is not discovered by scanning: each model package ships a one-line source file under `contentFiles/cs/any/` that NuGet adds to the referencing project. It holds a `[ModuleInitializer]` that calls the package's `Register()`, so the trimmer and NativeAOT see the model code and its embedded weights as reachable. `Register()` costs nothing; the weights are read on first use, once, and the model is shared by all callers (it is safe to call from many threads). Text extraction reaches the span model only through this seam (`docs/extraction.md`), so `.Extraction` does not reference the model package. The Tier 1 + Tier 2 package (`RecursiveAI.OneContact.Models`, from 1.1) reuses the seam: a package registers more interfaces or a higher rank; the parsers do not change. ## The Tier 1 model `RecursiveAI.OneContact.Models.Tier1` holds one saved OneNLP pipeline (`weights/tier1.onenlp`, embedded): a fast-tier encoder (hash embeddings of each word's lower-cased text, case-preserving text, prefix, suffix, shape and its neighbours, then three window layers) shared by three softmax heads and an entity head: | Head | Column | Labels | |---|---|---| | `address-fields` | Tag | `B-`/`I-` plus the field label, or `O` | | `name-parts` | Pos | `B-`/`I-` plus the part label, or `O` | | `name-class` | Morph | `person`, `company`, `multiple` or `unknown` on every word | | `contact-spans` | entities | BILUO tags over the words, six labels, each twice: the plain label and the label with `:start` for the first field of a contact block | **The span head.** `Tier1Model.Register()` registers the head with the address and name models. Extraction uses it on plain text only: on real web pages it made records and addresses worse than the rules alone, so a web page is read by the markup and the rules whether or not the package is referenced (`docs/extraction.md`, "The span model on real pages"). The head works through `Tier1Model.Find` for anyone who calls it. `Tier1Model.Find(text)` runs the head once over the words of a text (the same words as above) and reads the entities it wrote: a span is the field; a span whose label ends in `:start` also puts its offset in `BlockStarts`. A block is what the data calls a record: one person's or one company's entry, and its first field marks where it begins, so one head answers both questions. A span's confidence is the mean probability of its words' tags. The head knows nothing of rules or of the page around the text; the caller decides what to keep. The encoder shape is written in `src/OneContact.Models.Tier1/Tier1Pipeline.cs`. **Words.** A word is a run of letters, digits and combining marks; every other character that is not white space is a word; a line break is a word (where the lines end tells a recipient line from a street line). The training data is cut the same way (`Tier1Tokenizer`). **Reading labels.** A span is a run of words with the same label; a `B-` tag starts a new span. A span's confidence is the mean probability of its words. The class is the vote of the words' class labels weighted by probability; line breaks and punctuation do not vote. **Weights.** The pack takes the file from the directory in `ONECONTACT_TIER1_WEIGHTS`, checks it against `weights.manifest.props` (name, size, SHA-256) and embeds it; a missing or different file fails the build with both hashes. Packing without the directory fails saying what to set. A development build has no weights, and `Tier1Model.Instance` then throws `InvalidOperationException` naming the package. Retraining replaces the file and the manifest row, with no code change. **Package contents.** The assembly with the embedded weights, `weights.manifest.props`, `PROVENANCE.md`, `THIRD-PARTY-NOTICES.md`, both licences, the registration file. No training or evaluation data. ## Training The recipe is `recipes/tier1` (a console project over OneNLP's public `pipeline.Train`; see its README). Everything that shapes the model is in `Tier1Pipeline.cs` and the constants at the top of `recipes/tier1/Program.cs`; seeds are arguments. A run writes `last.ckpt` and `best.ckpt` and resumes from the last checkpoint with the same arguments; chunks of at most 30 minutes continue the same run. - Address examples are the rendered United States and Canada sets (`docs/training-data-address-us-ca.md`), words as the sets supply them plus a word for each line break. - Name examples are the rendered name and company set (`tools/data/provenance/names-us-ca.md`); a part's edges cut words, the class goes on every word. - Examples are chosen from each file by a hash of their text, the same on every run, so a resumed run sees the same examples; dev examples come from the sets' dev files. Selection is on the dev score (the mean of address-field span F1, name-part span F1 and name-class word accuracy). No evaluation set is read by the recipe. - A training example for one head has no label for the others; the loss ignores words without a label. ## The contact-span head The span head was trained on the contact blocks of `tools/data/blocks` (rendered from our own templates and word lists: contact blocks between unlabelled page chrome, notes and signatures) with the encoder frozen, so the address and name heads are unchanged by it. It is used on plain text only; see `extraction.md`. Measured rows for the model, with their candidates and method, are kept in `docs/maintainers/models-tier1-measurements.md`. # The Tier 1 + Tier 2 model (`RecursiveAI.OneContact.Models`) > **From 1.1.** The model packages described here ship in 1.1, after OneNLP 1.0. Version 1.0 runs on rules and markup alone; the seam in core that registers a model (`ContactModels`) is part of 1.0. The same design as the Tier 1 package (see `models.md`): one fast-tier encoder shared by an address-field head, a name-part head, a name-class head and a contact-span head, trained with OneNLP's `Train` on OneModel. It covers the United States, Canada and every Tier 2 country. Rules run first; the model reads only what the rules leave open, and never replaces a rule hit. `ParseDetail` says per field whether a rule or the model decided it. - **Size.** About 23 MB of weights, embedded in the package. Load time and memory are reported with the results at each release. - **Resolution.** When both model packages are referenced, this one wins. The Tier 1 package never reads Tier 2 text; this package reads US and Canada text as Tier 1 does. Without a model package, a Tier 2 address the rules refuse stays refused, and the error names both packages. - **Country.** The model labels fields; it does not tell countries apart. A Tier 2 read takes the country from the hint, or from the country the rules had decided before they refused the address, and is otherwise refused as country-undetermined. A model read of text with no hint never displaces a Tier 2 rule reading, and is not made a United States or Canada address on a postal code alone (a German, Spanish or Czech postcode looks like a ZIP): it needs a state or province. A wrong country would be invented content, so the refusal is the decision. - **Names.** For Tier 2 texts the name heads read only what the rules refuse, except in the countries where the model's person reading is allowed to replace the rules' "person for want of company evidence" default (a confidence of at least 0.8). US and Canada names follow Tier 1. - **Dependent locality** (the settlement or sector between street and postcode) is a label of the address head and its own field and line in a model read. Where the rules read such a line themselves, the model is not asked. - **Single encode.** A name is encoded once for its class and part heads; the outputs equal those of two encodes. - **Training data and licences.** The default weights are trained on open national address registers mixed with OpenStreetMap addresses (© OpenStreetMap contributors, ODbL 1.0), national name lists, and company names from GLEIF, Companies House and SEC EDGAR. The weights ship `MIT OR Apache-2.0` with every source credited: see the package's `PROVENANCE.md` and `THIRD-PARTY-NOTICES.md`. A recipe that rebuilds a model from the permissive registers alone is `recipes/models`. - **Retraining.** `recipes/models` (train and span stages) is a console project over OneNLP's public `Train`. Measured rows (development splits, real-page sets, candidates) are kept in `docs/maintainers/models-superset.md`. # The accurate tier > **From 1.1.** The model packages described here ship in 1.1, after OneNLP 1.0. Version 1.0 runs on rules and markup alone; the seam in core that registers a model (`ContactModels`) is part of 1.0. `RecursiveAI.OneContact.Models.Accurate` is the opt-in model package for harder text. It carries the same four heads as the fast tier on one frozen encoder: OneNLP's distilled multilingual student (`RecursiveAI.OneNlp.Models.Multilingual.FrozenEncoder`, six layers, width 384, 113 languages). ## Using it Reference the package; it registers itself through a source file the package adds to your project (no reflection, no call from you). `AccurateModel.Instance` is the loaded model; `AccurateModel.Load(stream)` loads a retrained head file. - **Which package serves what.** With several model packages referenced, the fast tier (`.Models.Tier1`, `.Models`) keeps normalizing single addresses and names, and the accurate tier serves the contact-span model that reads running text for `ContactExtractor`. With the accurate tier alone it serves addresses, names and spans. This is `ContactModels.Register(..., normalizationRank: 0)`. - **Rules first.** Nothing changes about precedence: a rule hit is never overruled, and `ParseDetail` says whether a rule or the model decided each field. - **No model referenced.** An address the rules cannot decide fails with `address.model-required`, whose message names the packages to add, this one included. ## Cost The encoder is a transformer run on OneModel's CPU kernels, so the tier is slower and larger than the fast tier: about 140 MB of encoder files, about 0.8 GB resident once the encoder is loaded, and tens of milliseconds for a short text instead of fractions of one. Measured figures for maintainers are in `docs/maintainers/models-accurate.md`. Use the fast tier when you read single addresses and names in bulk. ## OneNLP version This package is built on OneNLP 0.4. A project that references it and a fast-tier package must resolve one OneNLP version; the fast tiers are retrained against 0.4 for that. ## Retraining `recipes/accurate` trains the heads with OneNLP's public `Train` over the frozen encoder, in resumable chunks on CPU: `run.sh` (address head), `names.sh` (name heads), `spans.sh` (span head). Its README lists the inputs. Provenance: `src/OneContact.Models.Accurate/PROVENANCE.md`. # Address training data, United States and Canada How the examples for the address-field head are made, what a line looks like, and what the labels mean. The tool is `tools/data/AddressUsCa`; where the data comes from and its licences are in `tools/data/provenance/address-us-ca.md`. ## Running it ``` tools/data/fetch-nad.sh # US DOT National Address Database, text file (8.5 GB), checked against pins/nad.pin tools/data/fetch-oda.sh # Statistics Canada Open Database of Addresses, ten files, checked against pins/oda.pin tools/data/fetch-tiger.sh # Census TIGER/Line 2025 address ranges for the regions NAD lacks, checked against pins/tiger.pin AddressUsCa count --nad --oda --out # usable records per region, sampling rates AddressUsCa render --nad --oda --out --seed 20261105 AddressUsCa check --out --sample 5000 [--offset N --misses ] AddressUsCa count --tiger --out --caps PR=80000,VI=24500,*=27000 --per-range '*=3,VI=16,AS=16,MP=16' AddressUsCa render --tiger --out --countries us --dataset address-us-tiger --seed 20261105 AddressUsCa check --out --sample 3000 --per-region 600 # rule-parser recovery per region ``` `check` with `--offset N` takes line N of every sampling step instead of the first (so a sample used for tuning and a fresh one for measuring share no example), skips the span checks, and with `--misses` writes every example the rules did not fully recover, with the missed and the extra fields. The registers go to `$ONECONTACT_DATA/train/` (default `~/onecontact/data/train`), never into the repository. `count` reads both registers once and writes `selection.tsv`: for each country and region the number of usable records and a sampling rate, chosen so that every region gets the same number of records (small regions are taken whole) and the total is near the target (1.3 M US, 380 k CA). `render` keeps a record when `hash(record id, selection salt)` is below its region's rate, drops a record whose address repeats one already kept, and writes the examples and `manifest.json`. ## Files `us.train.jsonl`, `us.dev.jsonl`, `ca.train.jsonl`, `ca.dev.jsonl`: JSON Lines in the pipeline's training format (one object per line; unknown properties are ignored by the reader). | Property | Meaning | |---|---| | `text` | The address as free text, one line or several (`\n`). | | `tokens` | `[start, end]` UTF-16 offsets. Letters, digits and combining marks form a token; every other non-space character is a token on its own, so a hyphen or apostrophe never joins words; tokens are also cut at every span edge. A model that reads these files must use the given tokens (or this rule): the pipeline's default tokenizer keeps `4-123` and `O'Brien` as one token. | | `entities` | `[start, end, "label"]`, flat, in order, not overlapping, each edge a token edge. Everything not labelled is outside (separators, punctuation). | | `spans.fields` | `[start, end, "street-address"]`: the delivery line, from its first labelled part to its last. | | `id`, `country`, `lang`, `layout`, `region` | Source record id (`nad:` or `oda:` plus the register's own id), `US` or `CA`, `en` or `fr`, the layout used, state or province. | Every letter and digit of `text` lies inside an `entities` span; the `check` command verifies this and the other span rules on every line. ## Labels The names are the ones the rule parser emits, so rule output and training labels compare directly. | Label | Covers | |---|---| | `recipient`, `organization` | A whole line of invented text (a title, "Attn:" or "c/o" included). | | `street.house-number` | `123`, `12B`, `W123N456`. | | `street.fraction` | `1/2` after a house number. | | `street.pre-directional`, `street.post-directional` | `N`, `Northwest`, `O` (Ouest), spelled out or abbreviated; the period of an abbreviation is outside the span. | | `street.name` | The street name, including words the register keeps in the name (`County Road 12`, `du Parc`). | | `street.type` | `St`, `Avenue`, `Boul`; before the name in French. | | `street.unit-designator`, `street.unit-number` | `Apt`, `#`, `Suite`, `APP`, `BUREAU` and the number or letters; the `4` of `4-123 Main St` is a unit number with the hyphen outside. | | `street.box-designator`, `street.box-number` | `PO Box 12`, `CP 12`, and `BOX 5` after a route or military unit. | | `street.route-designator`, `street.route-number` | `RR 2`, `HC 64`. | | `street.station-designator`, `street.station-name` | Canada Post `STN A`, `SUCC B`, `RPO`. | | `street.military-designator`, `street.military-number` | `PSC 1234`, `UNIT 8672`, `CMR`. | | `street.general-delivery` | `General Delivery`, `GD`, `Poste restante`. | | `urbanization` | Puerto Rico: the urbanization line (`URB LAS GLADIOLAS`, `Urbanización X`, or the bare name), one span. | | `locality` | The city, or `APO`, `FPO`, `DPO`. | | `admin-area` | State, province or territory, as a code or a name (`QC`, `Québec`, `Quebec`). | | `postal-code` | ZIP, ZIP+4, or a Canadian postal code with or without its space. | | `country` | `USA`, `United States`, `Canada`, `CAN`. | ## Layouts and variants `tools/data/templates/layouts.tsv` holds the layouts (USPS Publication 28 labels, Canada Post labels in English and French, and single lines with commas, without, with semicolons, with the province in parentheses); `names.tsv` the invented name parts; `oda-excluded-providers.tsv` the providers left out. A template names slots (`{city}{, |state}{ |zip}`); the punctuation before or after a slot is written only when the slot has content, so a missing ZIP removes its comma. The variants drawn per example are casing, abbreviated or spelled-out street types and directionals (from the Publication 28 and Canada Post rule tables), unit placement and designator, optional recipient, organisation and country, PO box, rural route, highway contract route, military and general-delivery lines, missing fields, ZIP+4, French civic forms (type before the name, a comma after the number, `St-`/`Ste-` for saint), and typing or OCR edits. The rates are in the manifest. ## Regions the National Address Database lacks Puerto Rico, the territories and five states have no (or few) rows in NAD. `--tiger` reads Census TIGER/Line county address-range tables (the `.dbf` inside each `*_addrfeat.zip`; `*_edges.zip` for American Samoa and the Northern Mariana Islands, which have no ranges) and the ZCTA to place relationship file, and makes one record per house number drawn inside an address range. Street names come from the Census with their abbreviations, split with the Publication 28 tables; the house number and the city are synthetic (see `tools/data/provenance/address-us-ca.md`). Puerto Rico uses `templates/layouts-pr.tsv`: Spanish street prefixes stay in the street name (Publication 28 appendix I section 51.1), and the urbanization line comes before the delivery line. Option `--caps` replaces the shared per-region cap of `count` with fixed caps, `--per-range` sets how many numbers are drawn from each range, `--countries` and `--dataset` name the output files and the manifest. # Address training data, Tier 2 group A (France, Belgium, the Netherlands, Luxembourg, Germany, Austria, Switzerland, Liechtenstein, Monaco) How the examples for the address-field head are made for these countries, and what is different from the United States and Canada (`docs/training-data-address-us-ca.md`, whose file format and label names this follows). The tool is `tools/data/AddressTier2A`; registers, licences and counts are in `tools/data/provenance/address-tier2-a.md`. ## Running it ``` tools/data/fetch-tier2a.sh # the registers, each file checked against tools/data/pins/*.pin AddressTier2A count --data --out [--countries fr,be,...] [--targets FR=310000,...] AddressTier2A render --data --out [--seed 20261105] [--dev-per-mille 20] AddressTier2A check --out # span rules on every line AddressTier2A sample --out --per-country 300 # what the library's rule parser does with a dev sample ``` `count` reads each register once and writes `selection.tsv`: usable records per region and a sampling rate that gives every region the same number of records (small regions whole), with the total near the country's target. `render` keeps a record when `hash(record id, selection salt)` is below its region's rate, drops an address that repeats one already kept (not for Monaco, which is drawn with replacement), assigns dev by `hash(record id, split salt) mod 1000 < 20` before rendering, and writes `.train.jsonl`, `.dev.jsonl` and `manifest.json` (seed, renderer version, pinned register files, per-file SHA-256 and counts, rates, variants drawn). The same inputs and seed give the same bytes. ## What is read from each register | Country | Fields used | Region (sampling stratum) | Language of the example | |---|---|---|---| | FR | house number, repetition index, street, commune, the postal (upper-case) locality, lieu-dit, postcode | département (971–976 for the overseas departments; the collectivities 975, 977, 978 and 98x are left out) | fr | | BE | house number, box, street and place in fr/nl/de, postcode | region (BRU, VLG, WAL) | nl in Flanders; fr or nl in Brussels; de where the register gives German names, else fr | | NL | house number, letter, addition, street, place, postcode | the first two postcode digits | nl | | LU | street, number, locality, postcode | commune | lb (street names in Luxembourgish, French or German; French-typed names are written number-first) | | DE | street, number, addition, municipality (North Rhine-Westphalia only) | Regierungsbezirk | de | | AT | street, number, postcode, Zustellort | federal state | de | | CH | street, number, ZIP label, canton | canton | de, fr or it from the street's type word, else the canton's main language | | LI | as CH | LI | de | | MC | none: synthesized | MC | fr | ## Labels The US/CA label set, with two additions for these countries: `sorting-code` (France's `CEDEX 08`) and `dependent-locality` (a French lieu-dit, a Monaco quarter). `street.type` is split from the name only where the type is a separate word in front of the name (French and Italian: `Rue`, `Avenue`, `Via`); Dutch and German types are part of the name (`Kerkstraat`, `Hauptstraße`) and stay inside `street.name`. A country code in front of the postcode (`F-75008`, `CH-8001`, `FL-9490`) is labelled `country` with the hyphen outside; Luxembourg's `L-` is labelled the same way (the structural layer's postcode pattern is the four digits alone). `street.unit-*` carries Belgian `bus`/`boîte` with the box, and the French, German and Austrian secondary-address words (`Bâtiment`, `Appartement`, `Wohnung`, `Top`); `street.box-*` carries `BP`, `CS`, `TSA`, `Postfach`, `Postbus`, `Case postale`, `B.P.`. ## Layouts and variants Number before the street in France, Monaco and Luxembourg's French-typed streets (`12 rue de la Paix`, `12, rue de la Paix`); after it elsewhere (`Hauptstraße 12`). The postcode line puts the postcode first (`75008 Paris`, `1234 AB Amsterdam`, `L-1234 Luxembourg`). Each example is a multi-line label or a single line with commas, spaces, dashes, semicolons or slashes. Drawn per example, with the rates in the manifest: casing (as in the register, upper, lower, and the postal service's own form: France's upper-case without diacritics, PostNL's upper-case locality), the second language of Brussels and German-speaking Wallonia (and a bilingual `Rue … / …straat` street in 3 % of Brussels examples), abbreviated French types (`av.`, `bd`, `ch.`) and German `…str.`, `ß`→`ss` (always in Switzerland and Liechtenstein), Dutch and Belgian number additions (`12A`, `12-2`, `12/3`, `bus 3`), French `bis`/`ter`, a recipient or organisation line from our own name parts (`tools/data/templates/tier2a-names.tsv`), a country line, a post-box line, `CEDEX`, a synthetic secondary-address line, a missing postcode or locality, typing slips, glued and doubled spaces. ## Two gaps, stated plainly * **Germany is one state.** No national open address register exists; the only permissive state file fetched is North Rhine-Westphalia's building references (`dl-de/zero-2-0`). Its streets and municipalities are real; it has **no postcodes**, so each address gets a synthetic one inside NRW's real two-digit ranges (a fixed hash of municipality and street; stated in the manifest). Labelling is unaffected; postcode-to-place agreement is not real. Other states (Hamburg, Berlin, Brandenburg, Sachsen and others publish under `dl-de/by-2-0` in GML or per-county files) are a follow-up. * **Monaco is synthetic**: about 40 well-known public streets, numbers drawn per street, the single postcode 98000. # Address training data, Tier 2 group B (United Kingdom, Spain, Italy, Portugal, Ireland, Malta, San Marino, the Vatican, Andorra, Jersey, Guernsey, the Isle of Man) How the examples for the address-field head are made for these countries. The file format, label names, splits and determinism are those of the other address tools (`docs/training-data-address-us-ca.md`, `docs/training-data-address-tier2-a.md`); this note covers what differs. The tool is `tools/data/AddressTier2B`; sources, licences and counts are in `tools/data/provenance/address-tier2-b.md`. ## Running it ``` tools/data/fetch-tier2b.sh # the sources, each file checked against tools/data/pins/*.pin AddressTier2B count --data --out [--countries gb,es,...] [--targets GB=300000,...] AddressTier2B render --data --out [--seed 20261105] [--dev-per-mille 20] AddressTier2B check --out # span rules on every line AddressTier2B sample --out --per-country 300 # what the library's rule parser does with a dev sample tools/data/verify-tier2b.sh # render twice, compare every byte, check, sample ``` `count` and `render` work as in group A: a per-region sampling rate gives every region the same number of records (small regions whole) with the total near the target, dev is `hash(record id, split salt) mod 1000 < 20` taken before rendering (same salts as group A), and the same inputs and seed give the same bytes. The tool shares the tokenizer, random generator, example type, span rules, delimited-text reader and rule-parser sample with the other address tools by compiling their files; it adds its own readers, lexicon and renderer. ## What is real and what is drawn | Country | Street, number | Locality, postcode | Drawn | |---|---|---|---| | GB | every named road of OS Open Names | the road's populated place; the postcode units nearest to the road in its 10 km tile and postcode district | house number (many low numbers, a tail to ~180), a letter after it (4 %), flats | | ES | Catastro portals: street (with its type code), number, postcode, municipality | as the portal | floors and doors, post-box lines | | IT | ANNCSU civic numbers (street, number, exponent, frazione, South Tyrol's German name) | comune name from the ISTAT list; the CAP from GeoNames by comune, picked by a fixed hash where a city has several | secondary-address lines | | PT | our vocabulary (`tier2b-streets.tsv`) | GeoNames postcode (seven digits) and place; the concelho as the post-office name in 40 % | street, house number | | IE | our vocabulary | GeoNames routing key and place | street, number, the last four characters of the Eircode | | MT | our vocabulary (Maltese and English) | GeoNames locality code and name (Maltese name in 30 %) | street, number, the four digits of the postcode | | SM, VA, AD | our vocabulary | GeoNames postcode and place | street, number | | JE, GG, IM | our vocabulary (shared) | GeoNames postcode area and parish/place | street, number, the inward part of the postcode | A drawn postcode part is valid in form and unreal in fact; the manifest counts `postcode:drawn`. In the United Kingdom the road/postcode pairing is a stand-in for an address register that does not exist openly: the pair is plausible, not a statement about which postcode the street's houses have. No Royal Mail PAF, OS AddressBase or Eircode data is used. ## Labels and layouts The group A label set: `street.house-number`, `street.type`, `street.name`, `street.unit-designator`, `street.unit-number`, `street.box-designator`, `street.box-number`, `dependent-locality`, `locality`, `admin-area`, `postal-code`, `country`, `recipient`, `organization`. * **Street type** is labelled where it is a separate word: after the name in English (`Church` `Road`), before it in Spanish, Catalan, Galician, Italian, Portuguese, Maltese and Andorran Catalan (`Calle` `Mayor`, `Via` `Roma`, `Rua` `Augusta`, `Triq` `ir-Repubblika`). Spanish `C/`, Italian `V.le` and similar are the type with the slash or period inside the word or after it. * **Number position:** before the street in the English-speaking countries and in half of the Maltese ones; after it in Spain (`Calle Mayor, 12`), Portugal, Italy, San Marino, the Vatican and Andorra, with a comma in most Spanish, Portuguese and Andorran examples and in about half of the Italian ones. An Italian exponent is written `12/A`, `12A` or `12 A`. * **Spanish floors and doors** (`3º B`, `Bajo`, `1.º Dcha.`) and Portuguese ones (`2.º Esq.`, `R/C Dto.`) are one `street.unit-number` span; Italian `Scala B`, `Int. 5` and English `Flat 3` have a designator span and a number span. * **Last line:** UK `Post Town`, optional county line, postcode (own line, or after the town on the same line); Ireland locality then Eircode; Spain `28864 Ajalvir`, with the province after it in 30 % (`(Madrid)`, `, Madrid` or its own line); Italy `00145 Roma RM` with the province sigla (`admin-area`) in three quarters; Portugal `1100-053 Lisboa`; Malta `Valletta VLT 1117`; San Marino, the Vatican, Andorra postcode first (`AD500 Andorra la Vella`, `AD-500` in 10 %). * **Languages:** Welsh and Gaelic second names of OS Open Names roads (`cy`, `gd`), South Tyrol's German names (`de`), the Catalan, Galician and Spanish forms of Spanish street types (`ca` in Catalonia, the Balearic Islands and the Valencian provinces at 35–60 %, `gl` in Galicia at 55 %), Maltese and English, Andorran Catalan with Spanish (6 %) and French (6 %) forms. * **Variants drawn** (rates in the manifest): casing (as the source writes it, UPPER, lower, and the postal form: upper-case locality), type abbreviations, recipient and organisation lines from our own name parts (`tier2b-names.tsv`), a country line, a post-box line, a missing postcode or locality, typing slips, doubled spaces, a flat or unit line. ## Gaps stated plainly * **Spain:** the Catastro does not serve the Basque Country or Navarre (their cadastres are separate), so those provinces are absent and Basque-language street names do not occur. About 12 % of the municipalities are fetched (917 of the feed's municipal files; chosen by a fixed hash of the municipality code); a street whose name carries a parenthesised article (`CL IGLESIA (DE LA)`), a number with letters, or an unlisted type code is left out. * **Italy:** the open civic-number file has no postcode. A city with several CAPs gets one by a fixed hash, so the street-to-CAP pairing there is not real (the manifest counts `postcode:drawn`). Comuni the GeoNames list does not name are left out (4 % of rows). * **Portugal, Ireland, Malta, San Marino, the Vatican, Andorra, Jersey, Guernsey, Isle of Man:** the street names are our own vocabulary, not a register. Ireland has no county (the postcode list has none) and an Irish-language form does not occur. The Portuguese national address base (Base Nacional de Moradas) is published on dados.gov.pt, which was not reachable from the build network, so it is not used. * **Northern Ireland** is absent: OS Open Names covers Great Britain. * **Structural check against the pinned layer** (information, in `rule-sample.md`): Great Britain, Jersey, Guernsey, the Isle of Man, Malta, Portugal, San Marino and the Vatican pass; Spain, Italy and Andorra examples without a province lack a required `AdminArea` (the data file requires it); Ireland's postcode is `UnexpectedField` because the pinned data file gives Ireland no postcode field. # Address training data, Tier 2 group C (Norway, the Czech Republic, Latvia, Estonia, Denmark, Sweden, Finland, Iceland, Mexico, Poland, Slovenia, Slovakia, Lithuania, Croatia, Hungary, Romania, Australia) How the examples for the address-field head are made for these countries. File format, label names, splits and determinism are those of `docs/training-data-address-tier2-a.md`; this page says what differs. The tool is `tools/data/AddressTier2C` (it shares the tokenizer, random generator, example type and span rules with the other address tools and the delimited-text reader with group A). Registers, licences and counts: `tools/data/provenance/address-tier2-c.md`. ## Running it ``` tools/data/fetch-tier2c.sh # the four registers, each file checked against tools/data/pins/*.pin AddressTier2C count --data --out [--countries no,cz,lv,ee] [--targets NO=250000,...] AddressTier2C render --data --out [--seed 20261105] [--dev-per-mille 20] AddressTier2C check --out ``` `count` and `render` work as in group A: a per-region sampling rate gives every region the same number of records (small regions whole), dev is `hash(record id, split salt) mod 1000 < 20` fixed before rendering, repeated addresses are dropped, and the same inputs and seed give the same bytes. ## What is read from each register | Country | Records used | Region (sampling stratum) | Notes | |---|---|---|---| | NO | road addresses (`vegadresse`) of Matrikkelen Adresse: street, number, letter, postcode, post office | the first two digits of the municipality number | Farm and cadastral addresses (`124/1`) are left out. The register writes post offices in capitals; the examples use title case, with the capitals as the "postal" casing variant. | | CZ | RÚIAN address places with a house number (`č.p.`): street, house and orientation number, postcode, municipality, part of municipality, Prague district | first postcode digit | A village address has no street; the part of the municipality stands where the street would (`Libhošť 1`). Registration numbers (`č.ev.`) are left out. Prague is addressed by its district (`Praha 1`). | | LV | existing buildings of VARIS, read from the register's own address text | municipality | `Brīvības iela 10`, building part `k-2`, farm names in quotes (`"Riņņi"`, written without them). Parish (`pag.`) is the dependent locality, municipality (`nov.`) the admin area. | | EE | valid ADS address versions with a postcode: short address, municipality, settlement, city district, county | county | Flat numbers (`9-12`, `9a/4`) are labelled as unit numbers. A farm name with its number (`Kase-3`) stays one street name. | ## Layouts and variants Street before number everywhere. Norway, the Czech Republic and Estonia put the postcode first on the last line (`0155 Oslo`, `110 00 Praha 1`, `48216 Jõgeva maakond`); Latvia puts it last (`Rīga, LV-1010`). The Czech postcode is written with the space in 90 % of examples, Latvia's `LV-` is part of the postcode span (and missing in 10 %, which the structural layer flags as an invalid format). Estonian rural addresses carry the municipality on its own line and may end with the county. Each example is a multi-line label or a one-line text with commas, spaces, dashes, semicolons or slashes. Drawn per example, with the rates in the manifest: casing, recipient and organisation lines from our own name parts (`tools/data/templates/tier2c-names.tsv`), a country line and a country code in front of the postcode, a post-box line, synthetic flat designators (`H0201` in Norway, `byt 5`, `dz. 5`), a missing postcode or locality, typing slips and doubled spaces. ## Mexico, Poland, Slovenia, Slovakia, Lithuania These five use the same tool, splits, label set and determinism. Registers, licences and counts: `tools/data/provenance/address-tier2-c.md`. ``` tools/data/fetch-tier2c1.sh # MX, PL, SK and the LT geodatabase, each file checked against tools/data/pins/*.pin python3 tools/data/fetch-si-rpe.py # SI: pages through the public WFS once and writes the pinned files python3 tools/data/convert-lt-gdb.py # LT: geodatabase to two plain tables (needs the pyogrio package as a file-format converter) AddressTier2C convert --data --countries pl,sk,si # one-off plain tables for the large or awkward registers ``` The plain tables (`/derived/-addresses.tsv.gz`) have one address per line: id, region, street, street kind, house number, suffix, flat, second number, locality, dependent locality, admin area, postcode. Their hashes are in the pins and the provenance file; the conversion is deterministic. | Country | Records used | Region (sampling stratum) | Notes | |---|---|---|---| | MX | DENUE establishment addresses with a street name, an outside number (or `S/N`) and a postcode, one zip per state | state | Many establishments share an address; one of each is kept. The register writes capitals without accents in street names; the examples title-case them (`Av. Miguel Hidalgo y Costilla 577`). Kind of street (`AVENIDA`) and of neighbourhood (`COLONIA`) are written out or abbreviated at render time. The neighbourhood is the dependent locality (`Col. Roma Sur`), the municipality the locality, the state the admin area. Post codes that lost their leading zero are restored. | | PL | PRG address points: street, house number, postcode, locality | voivodeship (first two digits of the municipality code) | `ul.` is added to a plain street name (78 %, "ulica" 8 %, none 14 %); `al.`, `pl.`, `os.` and the like stay as the register writes them. A village without street names is addressed by the village (`Gliny Małe 135`). The register gives the locality but not the post office town; the locality stands in for it. Flats are synthetic (`30/5`, `30 m. 5`, `30 lok. 5`). | | SI | RPE house numbers with their street, settlement and post office | municipality | Last line is the post office name (`2314 Zgornja Polskava`). A village without street names is addressed by the settlement. The letter after a number is written with or without a space (`11 c`, `11c`). | | SK | Register adries entrances in force: orientation number, street, building registration number, municipality, part, postcode | county | `Hlavná 12`, or with the registration number first (`Hlavná 15/12`) in 40 % of examples that have both. A building without a street link is addressed by the part of the municipality (or the municipality) and its registration number. The postcode is written `811 01` (85 %) or `81101`. | | LT | address objects with their street or village, number, building block, flat (from the room table) and postcode; the locality is read from the register's own address text | municipality | `Gedimino pr. 10-5, LT-01103 Vilnius`; flats `-5`, `, butas 5`, ` b. 5`, `/5`; building block `15 k. 7` or `15, korpusas 7`. A village address carries its own line `Akmeniškių k., Vilniaus r. sav.`. `LT-` is part of the postcode span (and missing in 10 %). | Recipient and organisation lines come from `tools/data/templates/tier2c-c1-names.tsv` (Polish, Slovak and Lithuanian family names are chosen so that a given name and a family name drawn apart never disagree in sex; invented, no register). Post box designators: `Apartado Postal`, `Skrytka pocztowa`, `Poštni predal`, `Poštový priečinok`, `Pašto dėžutė` and their short forms. ## Denmark, Sweden, Finland (with Åland) and Iceland ``` tools/data/fetch-nordic.sh # the Finnish and Icelandic registers and the GeoNames lists, each checked against tools/data/pins/{fi,is,nordic}.pin AddressTier2C count --data --out --countries dk,se,fi,is AddressTier2C render --data --out --countries dk,se,fi,is ``` | Country | Records used | Region (sampling stratum) | Notes | |---|---|---|---| | FI | DVV building addresses (dBASE table) with one street name, a house number (`12`, `12 B`, `12-14`) and a postcode GeoNames knows; post office by postcode | the 19 regions of the register, Åland included | Finnish-only and Swedish-only addresses in their own language; bilingual addresses in Finnish. The staircase letter is written in capitals after a space in 40 % of examples (`Kauppakatu 5 B`) and a flat number may follow it. Åland addresses use `AX-` as the code prefix and `Åland` on the country line. | | IS | Staðfangaskrá rows with a street or farm name and no plot, land or special name; letter kept as written | municipality number | A farm with no number is a street name alone (`Unusel, 561 Varmahlíð`), as for Latvian villages. | | DK | GeoNames postcode areas with a drawn street name and number | region | Synthetic streets. Flat designations as floor and side (`2. th.`, `st. tv.`) in 3 % of examples. | | SE | GeoNames postcode areas with a drawn street name and number | county | Synthetic streets. Postcode `123 45`, without the space in 10 %; apartment as `lgh 1101`. | The last line is `postcode place` in all four (`2100 København Ø`, `123 45 Stockholm`, `00100 Helsinki`, `101 Reykjavík`), with the country code in front in 7 % (`DK-2100`, `SE-123 45`, `FI-00100`, `IS-101`). The postal casing variant writes the place in capitals for Finland and Sweden as for Norway. Post box lines are `Postboks`, `Box`, `PL` (Finnish) or `PB` (Swedish in Finland), and `Pósthólf`. Name parts for recipient and organisation lines are in `tools/data/templates/tier2c-names-nordic.tsv` (the renderer reads every `tier2c-names*.tsv`), street vocabulary in `tier2c-streets-nordic.tsv`. Denmark, Sweden, Finland, Iceland, ## Group C-3: Croatia, Hungary, Romania, Australia ``` tools/data/fetch-tier2c3.sh # the five source files, each checked against tools/data/pins/{hr,hu,ro,au,au-abs}.pin AddressTier2C count --data --out --countries HR,HU,RO,AU AddressTier2C render --data --out --countries HR,HU,RO,AU AddressTier2C check --out tools/data/try-tier2c3.sh # count, render, check and show examples for HU,AU,RO into working directories ``` The options, splits and determinism are those above; these countries are drawn by `RendererC3.cs` (layouts, rates and the labelled pieces are the same kind as the others) from `tools/data/templates/tier2c3-names.tsv` (recipient and organisation parts) and `tier2c3-streets.tsv`. | Country | Records | Region (sampling stratum) | Notes | |---|---|---|---| | HR | every address of the INSPIRE Addresses file: street, number with letter, post office name and postcode, settlement | first two postcode digits | The street name includes its type (`Brežna ulica`, `Ulica Stjepana Radića`). A settlement that is not the post office goes on its own line (`dependent-locality`). Flats follow a slash (`12/3`, drawn). | | HU | **synthesized**: each GeoNames postcode and place with 42 streets drawn from our list | county | Hungary has no open address register. Numbers are drawn (`12.`, `3/A.`, the full stop belongs to the number span). Half the label layouts follow the mail form (city, street, postcode on separate lines), the rest put the postcode first. Recipients are written family name first. | | RO | street rows of the Posta Romana workbook for Bucharest (with sector) and for the localities over 50,000, and its village rows | Bucharest sector or county | Three numbers are drawn per row inside the register's range (steps of two), anywhere on a street whose number cell is empty. A village has no street: the village name (50 % with `Sat`) stands in the street place with a drawn number. `nr.` belongs to the house-number span; flats and blocks (`bl. 4, ap. 15`) are drawn. Types appear in full (`Strada`, `Bulevardul`) or short (`Str.`, `Bd.`); the register writes person-named streets surname first (`Mincu Ion`) and the examples keep that. 20 % of examples use the cedilla letters. | | AU | **synthesized**: each ABS suburb or locality with its postal area (SAL to POA correspondence, ASGS Edition 3), plus the GeoNames postcode, place and state pairs ABS lacks, with 14 streets drawn from our list per place | state | Number before street; units as `Unit 3, 45 Smith Street` or `3/45 Smith Street`; the "postal" casing is Australia Post's capitals with short street types (`45 SMITH ST`, `SYDNEY NSW 2000`); state as the abbreviation or the full name. | Croatia, Hungary and Romania also draw an ASCII variant (diacritics dropped, 7 %), as people type them. Counts, licences and hashes: `tools/data/provenance/address-tier2-c.md`. # Address training data for the challenger model (OpenStreetMap, ODbL) How the examples for a challenger address-field model are made from the OpenStreetMap address file of the published libpostal parser training set. The file format, tokens and span rules are those of the other address tools (`docs/training-data-address-us-ca.md`); this note covers what differs. The tool is `tools/data/AddressChallenger`; sources, licences and counts are in `tools/data/provenance/address-challenger-osm.md`. **This data is a separate lineage.** It is ODbL data. It never feeds the default models: it lives under its own folder (`/challenger/train/osm-lp-v1/`), every line carries `"set": "challenger-osm"`, and the manifest says `challenger`. ## Running it ``` tools/data/fetch-challenger.sh # the file, checked against tools/data/pins/challenger-osm.pin tools/data/run-challenger.sh count render check verify ``` `count` makes one pass over the file and writes `selection.tsv`: for each country the lines in the file, the lines that convert, and a sampling rate (the target divided by the usable lines, at most 1; the default target is 250,000). `render` makes a second pass and keeps a line when `hash(line id, selection salt)` is below its country's rate, drops a line whose tagged text repeats one already kept in that country, converts it, and writes the train and dev files and `manifest.json`. A line's id is `osm::`. Dev is `hash(id, split salt) mod 1000 < 20`, fixed from the id alone. The same input, seed (default 20261106) and selection give the same bytes; `verify` renders again and compares every file. ## The source file `formatted_addresses_tagged.random.tsv.gz`: one address per line, `language TAB country TAB tokens`. A token is `word/label`; `|/FSEP` is a field separator and `,/SEP` a comma. The labels are libpostal's component labels (documented in libpostal's README). The file is in random order. It holds every OpenStreetMap address rendered with address-formatting templates, with abbreviations, dropped components and typing damage applied by its authors; some lines are only a city or a street. The language column is `unk` when no language was assigned. ## Lines left out * A country outside the list: the United States and its territories (US PR GU VI AS MP), Canada, Mexico, Australia, the United Kingdom, Jersey, Guernsey and the Isle of Man, the Latin-script Tier 2 countries of the EU, Switzerland, Norway, Iceland, Liechtenstein, Andorra, Monaco, San Marino and the Vatican, and the French overseas departments (GP MQ GF RE YT). Greece, Bulgaria and Cyprus are outside the Latin-script tiers. * A line with a `category`, `near`, `world_region`, `country_region` or `island` token: these are search-query words and informal regions with no counterpart among the address fields. * A line with a word outside the Latin script (ASCII, Latin-1 to Latin Extended-B, Latin Extended Additional, combining marks, general punctuation and the numero sign), a word longer than 64 characters, more than 40 tokens, a text longer than 300 characters, or no field other than the country. * Nothing is repaired: label noise in the source (a street tagged as a venue, a postcode glued to a city) stays. ## Writing the text The source has tokens, not text, so the text is rebuilt: * A field separator is written, per example (drawn from the id), as a line break (40 %, layout `osm.lines`), a comma and a space (45 %, `osm.comma`) or a space (15 %, `osm.space`). A `,/SEP` is a comma in the first two layouts and a space in the third. The source does not record which was used. * Words are joined by one space, except: `.` `,` `;` `:` `)` `]` `!` `?` attach to the word before; `(` `[` and `#` attach to the word after; `/` joins letters or digits on both sides (`140/8`); `-` joins digits (`608-610`); `'` joins letters (`l'Église`); a quotation mark attaches inwards. * The tokens of a set are our own (`tools/data/AddressUsCa/Tokenizer.cs`): letters, digits and combining marks form a token, every other character is a token, and tokens are cut at span edges. * A span is a run of consecutive words with one source label and no separator between. Punctuation-only words at the edges of a run are left outside it, as is a final period or comma of the last word. Brackets and quotation marks stay in the span they belong to. ## Label mapping | Source label | Our label | Note | |---|---|---| | `house` | `recipient` | A venue or building name. | | `house_number` | `street.house-number` | | | `road` | `street.road` | The whole street, type and name together. The source does not separate them; this set does not either. | | `unit`, `level`, `staircase`, `entrance` | `street.unit-designator` + `street.unit-number` | Leading words without a digit are the designator (`Apt`, `#`, `Flat`) and the rest the number. A run that starts with a digit (`3rd Floor`, `6OG`), or has no digit at all (`Ground Floor`, `Rez-de-chaussée`), is all number. | | `po_box` | `street.box-designator` + `street.box-number` | The same split; a run with no digit (`Postfach`) is all designator. | | `postcode` | `postal-code` | | | `suburb` | `dependent-locality` | An unofficial neighbourhood. | | `city_district` | `dependent-locality.district` | A borough or district with an official role. Kept apart from `suburb` as the source keeps them apart. | | `city` | `locality` | | | `state` | `admin-area` | | | `state_district` | `admin-area.district` | A county or second-level division (and the province in Italy and Spain). | | `country` | `country` | | | `category`, `near`, `world_region`, `country_region`, `island` | none | The line is left out. | `street.road`, `dependent-locality.district` and `admin-area.district` are labels this set adds; a consumer that wants the register-based label set merges them (`street.road` to a street name and type, the two `.district` labels to their parents). `spans.fields` holds the `street-address` span from the first to the last `street.*` span when there is one. ## Known limits * The file is from 2017-03-04 and reflects OpenStreetMap of that date and the conventions of its authors; later mapping is absent. * Labels are the source authors', assigned when the file was built; they are noisy (see above). Evaluation on the real sets will show how much. * A drawn layout is not a record of the original punctuation; connectors are joined by a fixed rule. * Addresses in OpenStreetMap include venue names; some are names of businesses or people. They are public map data, kept only for training and not reproduced in any report. # Architecture notes Normative notes on how the code and its data fit together. Behaviour of each contact kind (key rules, rendering rules, label schemes) gets its own document here when the code for it lands. ## Layout | Path | What | |---|---| | `src/OneContact` | `RecursiveAI.OneContact`: the value types, `Contacts`, parse results, keys, JSON converters and every kind's parsing. Base class library only. | | `src/OneContact.Phones.Geo` | Geocoder, carrier and time-zone mappers. References core. | | `src/OneContact.Extraction` | Extraction of structured markup into person and organization records (`docs/extraction.md`). References core and OneHtml. | | `src/OneContact.Models.*` | The model packages (shipping from 1.1): they stay in the solution but are not packed or published with 1.0. | | `tests/OneContact.Tests` | xUnit v3 tests. | | `tools/Ingest`, `tools/Ingest.` | The ingest harness and one converter per upstream source. | | `tools/fetch` | Fetch scripts: exact files at a pinned ref into a cache outside the repository. | | `tools/Oracles` | Drivers for reference implementations used as black boxes by parity tests. | | `upstream/` | Each source's `PIN` and the licence and notice files as upstream publishes them. | | `samples/OneContact.AotSmoke` | NativeAOT console app; CI publishes it with warnings as errors and runs it. | | `bench`, `fuzz` | BenchmarkDotNet rows; the fuzz host. | Build output (`bin`, `obj`, packages) never goes in the source tree. See the comment at the top of `Directory.Build.props`: `ONECONTACT_ARTIFACTS`, a shared build area found next to the checkout, or `artifacts/`. ## Value types `PhoneNumber`, `EmailAddress`, `WebAddress`, `PostalAddress`, `PersonName` and `CompanyName` are `readonly partial struct`s with the same shape: - `Canonical` (faithful to the standard or the authority's style), `Key` / `GetKey(KeyVersion)` (aggressive, versioned), `Flags` (a per-kind `[Flags]` enum named `Facts`), `Source` (a `SourceSpan` of UTF-16 offsets into the original input). - `Parse` and `TryParse` over `string` and `ReadOnlySpan`, `ISpanParsable` (the `IFormatProvider` is ignored: nothing depends on culture), `ISpanFormattable` (format `C` or none: the canonical form), `IEquatable`, `==` on canonical forms. - A low-level `OperationStatus TryParse(span, [Region,] out T value, out ParseDetail detail)` that never throws. - `default(T)` is the empty value: empty canonical form and key, empty source span. - A `[JsonConverter]` attribute pointing at a reflection-free converter that writes the canonical form as a string, and an entry in the source-generated `ContactJsonContext`. `Parse` throws `ContactParseException`, a `FormatException` carrying `Kind`, `Reason` (a `ReasonCode`) and `Span`. ### Reason codes A `ReasonCode` is a kind plus a number that is unique within the kind. Numbers are never reused or renumbered. Codes shared by every kind live in `Parsing/ReasonCode.cs` (kind `Unknown`); each kind adds its own as static members of the partial struct in its own file (`Phones/ReasonCode.Phone.cs`, named `ReasonCode.Phone...`). `Id` is `.`, such as `phone.region-required`. ### Keys Each kind has a static `Key.Compute(value, KeyVersion)` in the `Keys` namespace. A key-rule change ships as a new `KeyVersion`; a published version's output never changes, and its rules are written down here (one file per kind) with golden fixtures. ### Seams: who edits which files Each kind owns one folder under `src/OneContact/`: the struct, its parser (`Parser.cs`, an `internal static` class with one `TryParse` method), its key class, its facts enum, and its reason codes. The parser files currently hold a placeholder body (trim, keep the text), so the shared contracts have something to run on. Implementing a kind means replacing that parser body and growing the struct, key class and enum; no shared file needs to change. `Contacts.Normalize` calls the structs' `TryParse`, and `ContactKindDetector` (shape tests only) is the one shared place that decides a kind from text. ## Compiled data Upstream data becomes compact tables at build time and reaches the library as an embedded resource, read lazily. **Why an embedded resource and not generated C#:** a table of a megabyte or more as a C# array literal makes the compiler slow and the source diff unreadable; a resource is read with `Assembly.GetManifestResourceStream`, which the trimmer and NativeAOT treat as plain data (no reflection over types), and nothing is read until a table is asked for. **Format "OCDT" version 1** (little-endian): ``` offset 0 "OCDT" offset 4 u16 format version (1) offset 6 u16 reserved (0) offset 8 u32 table count N (at most 4096) offset 12 N directory entries: u8 name length (1..255), name (UTF-8), u32 offset from file start, u32 length ... table bytes ``` Tables are written in ordinal name order, so the same inputs give the same bytes. The reader validates the whole directory before handing out anything: bad magic or version, a count over the limit, truncation, duplicate names, and any table outside the file all raise `InvalidDataException`. What is inside a table is up to its source; each source documents its own table layouts here when it lands. **Wiring a source into a project.** A source needs `upstream//PIN` plus licence files, `tools/fetch/.sh`, a converter project `tools/Ingest.` (an `Exe` whose `Main` calls `IngestRunner.Main(args, new MySource())`), and one props file in the consuming project, for core `src/OneContact/Data/.ingest.props`: ```xml ``` The core project imports `Data/*.ingest.props`, so adding a source never edits a shared file. The tool is restored and built by the target itself, not referenced as a project (a reference would copy the tool into the output and publish folder of everything that uses the library); a fresh checkout needs no extra step. Before compiling, `build/Ingest.targets` builds the tool, runs it over the fetched files (`$ONECONTACT_UPSTREAM/`, or the `Files` metadata), and embeds `.ocd` as the resource `OneContact.Data..ocd`. The notice text and the provenance record are packed under `notices/` in the package. In code: ```csharp private static readonly EmbeddedTables Tables = new(typeof(Marker).Assembly, "OneContact.Data.phonenumber.ocd"); ReadOnlyMemory regions = Tables.GetTable("regions"); // first call reads and validates the file ``` `tests/OneContact.Tests` uses a made-up `synthetic` source the same way, which is what proves the path. ## Ingest procedure - A **pin** (`PIN`) is a text file: `source`, `licence` (SPDX), `repo`, `ref` (tag or full commit), `urlbase`, and one `file ` line per file. Paths are plain relative paths. - `tools/fetch/fetch-pin.sh ` fetches each file by exact path (never a clone or archive) into the cache and checks every hash. A mismatch fails and the file is not kept. - `tools/fetch/all.sh` runs `fetch-pin.sh` for every `upstream/*/PIN`; CI runs it before building, with `ONECONTACT_UPSTREAM` pointing at a temporary folder. A new source needs no registration there. - The harness (`IngestRunner`) checks every hash again before it converts anything, then writes `.ocd`, `.provenance.json` (source, licence, repo, ref, tool version, the path, hash and size of each input, and the hash and size of the output) and `.notice.txt`. Exit codes: 0 ok, 2 file missing or hash differs, 3 bad pin, 64 bad arguments. - Conversion must be deterministic: no clocks, no random numbers, no dependence on dictionary order. Rerunning on the same inputs gives byte-identical outputs; files whose content is unchanged are not rewritten. - Notices that must ship (Apache-2.0 for the libphonenumber and libaddressinput data, MIT for the libpostal dictionaries, MPL-2.0 for the Public Suffix List, CC0 credit lines for the legal-form list and the disposable-domain list) are listed in `THIRD-PARTY-NOTICES.md`, which every package carries. # Tool environment The scripts and tools under `tools/`, `recipes/` and `bench/` keep everything large outside the repository. Each location is an environment variable with a default under `~/onecontact/`; nothing in the repository names a machine or a private folder. | Variable | Default | Holds | |---|---|---| | `ONECONTACT_DATA` | `~/onecontact/data` | Downloaded registers and name lists (`train/`), rendered training sets (`rendered/`, for example `rendered/tier2-a`, `tier2-b`, `tier2-c-north`, `tier2-c-south`, `tier2-c-nordic`, `tier2-c-east`, `tier2-au`), challenger data (`challenger/`). | | `ONECONTACT_WORK` | `~/onecontact/work` | Scratch for runs: bench snapshots and results (`bench/`), quick trial renders (`tier2-east/`), saved models (`models/tier1.onenlp`). | | `ONECONTACT_EVAL` | `~/onecontact/eval/private` | Evaluation sets (`tier2/group-a`, `group-b`, `c-north`, `c-south`, each with `sets/`; `sets/` for US and CA; `fresh/names`, `fresh/ro-mx`, `fresh/addresses` for the sets drawn for the 1.0 gate). Private: they hold personal data and stay on the machine that holds them. | | `ONECONTACT_ORACLES` | `~/onecontact/oracles` | Pinned oracle installs (jars, JDK, Python environments). | | `ONECONTACT_UPSTREAM` | `.upstream/` in the checkout | Upstream data files fetched by `tools/fetch/all.sh`. | | `ONECONTACT_ARTIFACTS` | `artifacts/` in the checkout | Build output. | | `ONECONTACT_LOCAL_FEEDS` | unset | Extra NuGet sources (semicolon separated) for Recursive AI preview packages. | | `ONECONTACT_TIER1_MODEL`, `ONECONTACT_MODELS_MODEL`, `ONECONTACT_WEB_MODEL`, `ONECONTACT_TIER1_WEIGHTS` | unset | Model files for tests, evaluation and packing. | | `ONECONTACT_TIER2_OUT`, `ONECONTACT_TIER2B_OUT`, `ONECONTACT_TIER2B_DATA` | under `~/onecontact/` | Output and input folders of the recovery tools. | Scripts assume `dotnet` (.NET 10 SDK) is on `PATH`. # Changelog All notable changes are recorded here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow [Semantic Versioning](https://semver.org/). ## [1.0.0] - 2026-11-07 First release. All packages share one version. `KeyVersion.V1` is frozen with this release: from here on, any change to a key rule ships as a new version. ### Added **`RecursiveAI.OneContact` (the .NET base class library only)** - Value types `PhoneNumber`, `EmailAddress`, `WebAddress`, `PostalAddress`, `PersonName` and `CompanyName`, and `Region`. Each has `Parse`/`TryParse` on strings and spans, a canonical form, a versioned comparison key (`KeyVersion`), flags, and offsets into the input. They implement `ISpanParsable`, `ISpanFormattable` and `IEquatable`, and ship System.Text.Json converters and a source-generated `ContactJsonContext`. `ParseDetail`, reason codes (`ReasonCode.Id` and `ReasonCode.Value`) and the `OperationStatus`-returning low level are available for every type; `Contacts.Normalize` and `Contacts.SameKey` work across types. - Phones for every region in libphonenumber's metadata: parse, validate, possibility, number type, E.164, international, national, RFC 3966, out-of-country and mobile-dialing formats, `AsYouTypeFormatter`, a matcher with leniency levels, short numbers with their cost classes, and example numbers. A `PhoneNumberParser` loads user-supplied newer metadata. Results equal libphonenumber 9.0.43 on generated cases for every region and operation. - Email: RFC 5322 and RFC 6531 addresses with display names and obsolete forms, IDNA (UTS #46) domains, provider-aware keys (Gmail, Outlook, Yahoo, Fastmail), role, free-provider and disposable flags, and typo suggestions that are never applied. - Web addresses: the WHATWG URL Standard and the Public Suffix List (shipped unmodified), tracking parameters removed from the canonical form, a missing scheme assumed as `https` and flagged, social-profile flags, `IsWebsite`. - Addresses: a structural layer for every region in libaddressinput's data (fields, layout, required fields, postcode patterns, `Validate()`); parsing, standardization to each postal authority's published style (`AddressStyle.Postal` and `AddressStyle.Display`, single-line and multi-line), structural validation and libpostal-style expansions for the United States, Canada, Mexico, Australia, the United Kingdom with Jersey, Guernsey and the Isle of Man, and the Latin-script countries of Europe (Austria, Belgium, Croatia, Czechia, Denmark, Estonia, Finland, France, Germany, Hungary, Iceland, Ireland, Italy, Latvia, Liechtenstein, Lithuania, Luxembourg, Malta, Monaco, the Netherlands, Norway, Poland, Portugal, Romania, San Marino, Slovakia, Slovenia, Spain, Sweden, Switzerland, Andorra and the Vatican). Country inference from the text when no country is given; `address.country-undetermined` when the text does not say. `PostalAddressJsonConverter(Region)` reads strings as free text of a country. - Person names (titles, suffixes, particles, second family names, name order, several people in one string, casing restoration traced to offsets) and company names (ISO 20275 legal forms with ELF codes, DBA and c/o, a leading "The"). Nothing is inferred about gender, origin or salutation. - Czech, Slovak and Croatian settlement lines between the street and the postcode line are read as the dependent locality, and a Bucharest sector as the dependent locality of the town; an addressee written in front of the street name without a comma (Hungarian and Nordic forms) is left to the model rather than read as part of the street name. - `ContactModels`, the seam through which a model package registers itself (the packages follow in 1.1); `ParseDetail` says per field whether a rule or a model decided it. **`RecursiveAI.OneContact.Phones.Geo`** - `PhoneGeocoder`, carrier and time-zone mappers, with the original-carrier caveat in their documentation. **`RecursiveAI.OneContact.Extraction`** - `ContactExtractor` over HTML (string or bytes), plain text, or a set of pages from one site that you supply (it never fetches). Markup readers for JSON-LD, microdata, RDFa Lite, h-card, `mailto:`/`tel:` links and `
`; visible-text reading through OneHtml's cascade; decoding of obfuscated emails with a `Deobfuscated` flag; `Person` and `Organization` records with their flat fields, sources, byte offsets into the original HTML, confidence, normalized values and kept disagreements; grouping and key-based de-duplication; site merge; `ContactExtractor.Limits`. ### Notes - Version 1.0 runs on rules and markup alone: `RecursiveAI.OneContact`, `RecursiveAI.OneContact.Phones.Geo` and `RecursiveAI.OneContact.Extraction`. Text the rules cannot read is refused with a reason, and nothing is read in a wrong country or printed that the text does not contain. - The trained model packages (`RecursiveAI.OneContact.Models.Tier1`, `RecursiveAI.OneContact.Models` and `RecursiveAI.OneContact.Models.Accurate`) ship in 1.1, after OneNLP 1.0. The `ContactModels` seam in core, which they register through, is part of 1.0. - Everything runs offline. No package opens a socket, resolves a name or downloads a file. - `CultureInfo` never affects a result. - A hinted German, Spanish, Croatian or Mexican address whose postcode also exists in France's overseas range (97xxx, 98xxx) is read in the hinted country. Without a hint, a 97xxx or 98xxx postcode with a German or Mexican cue is not read as French. - Without a hint, the United States and Canada rules read a text only when it shows the country (a state or province the country has, a ZIP+4, a Canadian postal code, or the country written out). A bare five-digit number after a word, `N.L.` or `B.C.` with a five-digit postcode, and `Baja California` are no longer read as American or Canadian. A Canadian postal code outweighs a five-digit house number, and `6980 LA ROCHE` is not a Dutch postcode.