--- title: "Third-party notices" url: https://recursiveai.org/onenlp/docs/third-party-notices/ description: "OneNLP's own code is licensed MIT OR Apache-2.0, Copyright (c) Recursive AI, LLC." version: "1.0.0" source: https://github.com/recursive-ai-oss/onenlp-csharp/blob/59bd14caa0e08a13494d74e29c396d1444730911/THIRD-PARTY-NOTICES.md --- # Third-party notices OneNLP's own code is licensed `MIT OR Apache-2.0`, Copyright (c) Recursive AI, LLC. Packages depend on other Recursive AI libraries (`RecursiveAI.OneFont`, `RecursiveAI.OneModel`, `RecursiveAI.OneModel.Training`), which carry the same licence and their own notices. Each third-party dataset, model or weight file that is packed is listed here with its licence and provenance. ## Language-identification model (packed in `RecursiveAI.OneNlp`) `LanguageId/langid.bin` is trained by `tools/OneNlp.LangId` on text from the following sources, credited as their licences require. The model is a table of numbers derived from the texts; it contains no sentence. Its provenance log is `recipes/langid/PROVENANCE.md`. - **Leipzig Corpora Collection** (Wortschatz Leipzig, Leipzig University; D. Goldhahn, T. Eckart, U. Quasthoff, "Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages", LREC 2012), news, web and Wikipedia text corpora, offered for download under Creative Commons Attribution 4.0 (https://creativecommons.org/licenses/by/4.0/). The list of corpora is `tools/fetch/langid/sources.tsv`. - **Tatoeba** (https://tatoeba.org), sentences contributed by the Tatoeba community under Creative Commons Attribution 2.0 France by default (https://creativecommons.org/licenses/by/2.0/fr/); individual sentences may carry other licences. - **Wikipedia** (Southern Sotho, Tswana and Tsonga editions, and the Wikipedia text inside the Leipzig corpora), Creative Commons Attribution-ShareAlike 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), by the Wikipedia contributors. The training data of this model includes share-alike text. - **FineWeb** (Hugging Face, sample-10BT of the English edition), released under the Open Data Commons Attribution License 1.0 (https://opendatacommons.org/licenses/by/1-0/) and subject to the Common Crawl terms of use, and **English Wikipedia** (20231101 dump), Creative Commons Attribution-ShareAlike 3.0 and GFDL, by the Wikipedia contributors. Sentences of this text, labelled by an earlier English model of this project, were part of the training data of the English tagging and entity models (`recipes/en/selftrain`). The training data of these models includes share-alike text. Packages depend on other Recursive AI libraries (`RecursiveAI.OneFont`, `RecursiveAI.OneModel`, `RecursiveAI.OneModel.Training`), which carry the same licence and their own notices. `RecursiveAI.OneNlp.Models.English` ships weights we trained on CC BY-SA 4.0 treebank and NER data; its notices are in that package (`src/OneNlp.Models.English/THIRD-PARTY-NOTICES.md`). `RecursiveAI.OneNlp.Models.Embeddings` redistributes the MIT-licensed `BAAI/bge-small-en-v1.5` embedding model, converted to ONNX; its notices are in that package (`src/OneNlp.Models.Embeddings/THIRD-PARTY-NOTICES.md`). `RecursiveAI.OneNlp.Models.Embeddings.Multilingual` (with three Shard packages that carry byte ranges of the file) redistributes the MIT-licensed `intfloat/multilingual-e5-small` embedding model, converted to ONNX; its notices are in that package (`src/OneNlp.Models.Embeddings.Multilingual/THIRD-PARTY-NOTICES.md`). No other third-party models or weights ship; each addition will be listed here with its licence and provenance before it is packed. The data it does ship is listed under "Data packed with the library". ## Data packed with the library - `src/OneNlp/Recognizers/Data/shared/sequence.rules` lists the domain endings (top-level domains) of the IANA Root Zone Database, `tlds-alpha-by-domain.txt` version 2026110304 (https://data.iana.org/TLD/tlds-alpha-by-domain.txt). IANA dedicates its registry data to the public domain under CC0 1.0 (https://www.iana.org/help/licensing-terms). - `src/OneNlp/Recognizers/Data/en/currency.rules` lists the alphabetic currency codes of ISO 4217. The codes are facts of the published standard (checked against List One as published by the maintenance agency, dated 2026-09-17); the file carries no text of the standard. - `src/OneNlp/Recognizers/Data/en/holidays.rules` lists the dates of holidays. The United States federal holidays follow 5 U.S.C. 6103(a) (federal law, public domain). The bank holidays of England and Wales follow the dates published at https://www.gov.uk/bank-holidays: contains public sector information licensed under the Open Government Licence v3.0 (https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/). The other observances are customary dates, and the file carries no text of any source. - `src/OneNlp/Tokenizers/Hf/NormalizationData.g.cs`, `ScriptData.g.cs` and `BertCleanData.g.cs` are generated from the Unicode Character Database 17.0.0 (`UnicodeData.txt`, `DerivedNormalizationProps.txt`, `DerivedAge.txt`, `Scripts.txt`), © Unicode, Inc., under the Unicode License v3 (https://www.unicode.org/license.txt): permission to deal in the Data Files without restriction, provided the copyright and permission notice appear with them. ## Distilled encoder (`student-v1`) The distilled multilingual encoder (`recipes/accurate/student-v1/PROVENANCE.md`) was trained to reproduce the hidden states of **mmBERT-base** (Marone et al. 2025; MIT licence, JHU CLSP) on text from **FineWeb-2** and **FineWeb** (Hugging Face; ODC-By 1.0, subject to the Common Crawl terms of use) and **Wikipedia** (the Wikipedia contributors; CC BY-SA 3.0 and GFDL). Its token table is initialised from mmBERT's, and it uses mmBERT's tokenizer. The training data includes share-alike text. ## Open-type entity recognizer (`recipes/openner`) The open-type entity head (`recipes/openner/PROVENANCE.md`) was trained on **Few-NERD** (Ding et al. 2021; CC BY-SA 4.0, text from Wikipedia, CC BY-SA), on **Universal NER** English-EWT (Mayhew et al. 2024; CC BY-SA 4.0) and on entity labels written by **Qwen2.5-7B-Instruct-AWQ** (Alibaba Cloud; Apache-2.0, run once to label passages; its weights are not redistributed) for passages of **FineWeb** (Hugging Face; ODC-By 1.0, subject to the Common Crawl terms of use) and **English Wikipedia** (CC BY-SA 3.0 and GFDL). The training data includes share-alike text. **CrossNER** (Liu et al. 2021; MIT licence, text from Wikipedia) is used only to measure the model and is not part of any package. ## Test data (not packed) `tests/testdata/unicode/WordBreakTest.txt` and `SentenceBreakTest.txt` are the Unicode 17.0.0 UAX #29 conformance files, © 2025 Unicode, Inc., redistributed under the Unicode License v3 (https://www.unicode.org/license.txt). They are used only by the test project and are not part of any package. ## Test fixtures `tests/OneNlp.Tests/Matching/expected.json.gz` records the output of spaCy 3.8.18 (MIT licence, Copyright Explosion) on documents and patterns written for this repository. It is used only by tests. `tests/OneNlp.Tests/NounChunks/cases.json.gz` records the noun chunks spaCy 3.8.18 (MIT licence, Copyright Explosion) returns on randomly generated trees and on the tree shapes (heads, relations and parts of speech, with placeholder words) of the Universal Dependencies English-EWT development set (CC BY-SA 4.0, credit: Universal Dependencies English Web Treebank). It is used only by tests. `testdata/accurate/` holds randomly initialised tiny encoders (a two-layer BERT-style one and a four-layer ModernBERT-style one, width 16) exported to ONNX, a hand-written WordPiece `tokenizer.json` and PyTorch hidden states for fixed ids, all generated by `tools/accurate/make_tiny_fixture.py` and `make_tiny_modernbert_fixture.py`; `tiny-student.*` is a random two-layer encoder with a factorised embedding from `tools/distill/make_tiny_student_fixture.py`. No released model or third-party text is in it. - `src/OneNlp/Recognizers/Data/{es,pt}/datetime.rules` list the dates of holidays: Spain (Estatuto de los Trabajadores art. 37.2), Mexico (Ley Federal del Trabajo art. 74), Brazil (Leis 662/1949, 10.607/2002 and 14.759/2023) and Portugal (Código do Trabalho art. 234), all public law, and the Christian feasts counted from Easter. The other observances are customary dates, and the files carry no text of any source. - `src/OneNlp/Recognizers/Data/{it,nl}/datetime.rules` list the dates of holidays: Italy (Legge 27 maggio 1949, n. 260, as amended) and the Netherlands (Algemene termijnenwet art. 3), both public law, and the Christian feasts counted from Easter. The other observances are customary dates, and the files carry no text of any source. - `src/OneNlp/Recognizers/Data/zh/datetime.rules` lists the dates of Chinese holidays: the statutory ones follow the Measures for National Holidays (State Council Order, 2013) and the State Council's yearly notices, and the lunar festivals for 2022-2030 are pinned dates taken from the published calendar (2028-2030 from the astronomical calendar rather than a notice). They are facts of public law and the calendar; the number and date conventions follow Unicode CLDR as facts (Unicode License v3), with no CLDR file shipped. - `src/OneNlp/Recognizers/Data/{tr,hi}/datetime.rules` list the dates of holidays: Turkey (Law 2429 on national and public holidays; the dates of Ramazan and Kurban Bayramı from the lists of the Presidency of Religious Affairs, Diyanet) and India (the Republic Day, Independence Day and Gandhi Jayanti dates, and the gazetted and restricted holiday lists of the Department of Personnel and Training for 2026 and 2027), all facts about dates, and the Christian feasts counted from Easter. The files carry no text of any source. ## Ligurian Genoese Ligurian Treebank (GLT) The base-size multilingual model was trained on the 20-sentence training split of **UD Ligurian-GLT r2.18**, contributed by **Stefano Lusito and Jean Maillard**, under the Computational Use of Data Agreement 1.0 (C-UDA). Source and acknowledgments: . The provider acknowledges De Ferrari Editore and the newspaper O Stafî. Only computational use was made; no treebank text is redistributed with the model. C-UDA §2.2: “The C-UDA does not impose any restriction with respect to the use, modification, or distribution of Results.” The training data retains its own terms. ## Hindi hand-annotated sentences (`testdata/hi`) `testdata/hi/selfdev.conllu` (100 sentences) and `testdata/hi/nerspot.iob2` (91 sentences): 17 of the first and all of the second are Hindi web sentences from FineWeb-2 (Hugging Face, `HuggingFaceFW/fineweb-2`, Open Data Commons Attribution License 1.0, subject to the Common Crawl terms of use); the other 83 sentences of the first, and `selfdev.lemmas.tsv`, are text we wrote. All part-of-speech tags, dependency relations, lemmas and entity labels were written by us. They are used to check the Hindi model, never to train it.