What oneyaml-csharp is measured against, the latest results, and every case it is known not to pass.
The library targets YAML 1.2 with the Core schema. It is scored against the official yaml-test-suite, release data-2022-01-17, plus a set of cases written for the oneyaml family that cover edits, patches, writing, safety and diagnostics. Both run through the conformance kit of the oneyaml family, oneyaml-kit, which drives the library through its public API: YamlDocument.Parse and its nodes for events and values, YamlSerializer for writing, and ApplyPatch and ApplyMergePatch for patches. The kit holds the cases, the corrected expectations and the adapter for this library. From a checkout of the kit, run bin/kit fast --adapter oneyaml-csharp with KIT_ONEYAML_CSHARP_SRC set to a checkout of this repository.
The tests in this repository that run the suite and the kit's cases through the library also run in CI, on every push to main and every pull request, against a pinned kit revision, so a change that breaks a case fails CI. Locally they are skipped unless ONEYAML_KIT names a kit checkout.
Results
The line below is generated by the kit's report tool from its result file for this library; it is not edited by hand. It gives:
- the yaml-test-suite release and the pass counts against it, loose and strict, out of every case of that release;
- for the cases where the kit corrects the upstream expectation, the pass counts against the corrected expectations, apart from the upstream counts;
- the pass counts on the kit's own cases, and whether the slow cases were run.
The tool refuses to write the line from a run that left cases out or that has a failure not on the deviation list below. To run the same check yourself, use the kit command above against a checkout of the release tag.
yaml-test-suite data-2022-01-17: 680/681 loose, 586/681 strict. Against the kit's corrected expectations for 12 of them: 12/12 loose, 1/12 strict. Kit's own cases, all of them including the slow ones: 609/609 loose, 609/609 strict. Details
How to read the counts
- Loose counts the outcome only: valid input must give exactly the expected events, value or bytes, and invalid input must be reported as an error.
- Strict counts everything the case states. For invalid input that includes the error category and position, and the events emitted before the error.
- Corrected counts judge the cases where the upstream expectation is wrong against the kit's corrected one. The kit lists the corrections and their reasons.
- The kit's own cases are counted apart from the suite. A run of the fast set leaves out the slow cases, and the line says so.
Known deviations
The cases the library does not pass, each with the expectation it was judged against (upstream or corrected), the mode (loose or strict), the outcome and the reason. The list is generated from the kit's deviation file for this library. The kit fails a run if a case fails that is not on the list, and also if an entry on the list starts to pass, so the list cannot go stale unnoticed.
2 known deviations:
| Case | Expectation | Mode | Verdict | Reason |
|---|---|---|---|---|
yts:UGM3@value | primary | loose | fail | The upstream value reads 450.00 as the integer 450; the library reads the float 450.0, which is right under the core schema (the corrected expectation passes). |
yts:UGM3@value | primary | strict | fail | The upstream value reads 450.00 as the integer 450; the library reads the float 450.0, which is right under the core schema (the corrected expectation passes). |
What the results do not say
- The suite covers the grammar. It is not a benchmark.
- A pass is a pass of the kit's checks at the revision shown. It is not a proof of conformance for input the suite does not contain; the library is also fuzzed (
tests/OneYaml.Fuzz). - The kit's own cases were written for this project family. They are a stricter and broader check than the upstream suite alone, and they are the project's judgment of correct behaviour where the specification leaves a choice, such as where a new mapping key goes.
- The compatibility adapters are not scored by the kit. They are checked against the real YamlDotNet and SharpYaml by the differential tests in
tests/OneYaml.Compat.Tests, and their differences are listed in YamlDotNet differences and SharpYaml differences.