onecrypto sop and the interoperability scoreboard

onecrypto sop <subcommand> implements the Stateless OpenPGP Command Line Interface (draft-dkg-openpgp-stateless-cli, version 16): exit codes, -- handling, @ENV: and @FD: indirect inputs, and the Armor/dearmor rules of the draft. It is the harness the OpenPGP interoperability test suite drives, and a small command line in its own right.

Install and help

dotnet tool install --global RecursiveAI.OneCrypto.Cli --prerelease

onecrypto --help lists the commands and onecrypto sop --help the sop subcommands; onecrypto sop <subcommand> --help (or onecrypto sop help <subcommand>) shows one subcommand's options. The draft defines no help option, so this is a convenience that exits 0 and writes to standard output.

verify, inline-verify and decrypt --verifications-out end each VERIFICATIONS line with the optional JSON object of draft §7.10.1: {"signers":["alice.cert"]} names, as given on the command line, every CERTS argument that holds a certificate that could have issued the signature (an argument named twice is listed once; a certificate in two arguments lists both).

What works now

encrypt chooses the container from the recipients' keys (see openpgp.md): SEIPD v2 for keys that read it, LibrePGP OCB Encrypted Data for version 5 keys and keys announcing it, SEIPD v1 otherwise; --profile=rfc4880 forces SEIPD v1. decrypt also reads GnuPG 2.5's Kyber keys (X25519 and X448) and bzip2-compressed messages.

An algorithm the platform lacks (for example ML-DSA or ML-KEM on an OpenSSL without them) exits 13 (UNSUPPORTED_ASYMMETRIC_ALGO). The subcommands are covered by in-process tests (tests/OneCrypto.OpenPgp.Tests/SopTests.cs) and the two-way scripts in tools/oracles/ (gpg-messages.sh, rnp-messages.sh, sop-matrix.sh).

The scoreboard

The suite (OpenPGP interoperability test suite, pinned commit in tools/oracles/interop-suite.sh) is a black box: it is built, executed, and only its results are read. Every test has a producer and a consumer role that both go through sop. Every test has two roles, so a round-trip case can use our generate-key as the producer as well as the consumer of what Sequoia, GopenPGP and the test data produced.

Run it (on a machine with the oracles installed):

tools/oracles/interop-suite.sh                      # fetch, pin and build the suite (nettle and libclang user-locally)
tools/oracles/gosop.sh; tools/oracles/sequoia.sh    # driver oracles (the Sequoia sop driver is built by interop-suite.sh)
dotnet build -c Release src/OneCrypto.Cli
tools/oracles/interop-scoreboard.sh <path to onecrypto.dll> <output dir>
python3 tools/oracles/interop-summary.py <output dir>/results.json
python3 tools/oracles/interop-vs-sqop.py <output dir>/results.json   # cells Sequoia passes and we fail

interop-show.py and interop-dump.py show or extract one cell so the command can be repeated by hand.

The scoreboard script writes config.json for the drivers it finds, leaves out those that do not answer version, lists them in missing.txt, and writes results.json (the HTML report needs the suite's own working directory: run the suite with --json-in there).

Latest run (251 tests, 14 sections, one score per driver per case)

Run on the release tree, twice: on a platform with ML-KEM and ML-DSA (OpenSSL 3.5.9) and on one without (OpenSSL 3.0.13, as hosted CI has). The suite reports four scores here, Success, Failure, Neutral and Unknown; no IO score appears. "Unknown" is the suite's verdict when the producer chose another algorithm than the one under test (9 cells, below).

On the platform with ML-KEM and ML-DSA our driver generates the post-quantum profiles, so sqop and gosop each fail 36 more cells there, in which our post-quantum keys and messages are consumed by them (sqop's nettle build has no ML-KEM or ML-DSA at all; the gosop failures were not investigated). Our own 9 extra failures on the platform without ML-KEM and ML-DSA are the post-quantum message and signature cells, which exit 13 there.

Cells where Sequoia succeeds and we do not: 59 on both platforms, none unexplained. The platform without ML-KEM/ML-DSA adds no cell to this list.

Cells where Sequoia and GnuPG agree and we fail: none. GnuPG is not a driver of the suite, so each class was run against GnuPG 2.5.24 by hand on the suite's own artifacts (gpg --verify, --encrypt, --decrypt, --sign). Of the 36 + 9 secret-key cells the only ones GnuPG does not do are the 3 with a 1025-bit RSA key (primary key decryption, subkey decryption and primary key signing): its agent does not import that secret key ("No secret key"), a limitation of its key import, not a policy decision, while Sequoia refuses on policy and we perform the operation. We list them here instead of counting them as agreement.

The nine "Unknown" cells (Symmetric Encryption Algorithm support, production side, OCB with a version 5 key that lists IDEA, TripleDES, CAST5 or Blowfish): the suite asks for a message under an algorithm that cannot be used with OCB; we use AES-128 with OCB, sqop and gosop write SEIPD version 1 ("Producer did not use AEAD"), which the suite accepts.

The list documented before this run had 69 cells (60 RSA, 5 SHA-1, 4 ML-KEM). On the merged tree it first came to 72: the 3 version 5 cells above were new with the LibrePGP writer, and the 4 ML-KEM cells ("Encrypt-Decrypt roundtrip with v4 EdDSA/ECDH,ML-KEM-768+X25519 key") had been put down to the platform but also failed on OpenSSL 3.5.9. That was a defect, with another one beside it, both fixed in this run: Sequoia pads the wrapped ECDH value to 40 octets (21 padding octets for AES-128), which RFC 9580 §11.5 allows, and we accepted 8 octets of padding at most; and we encrypted to RSA keys under the policy's size for new data (RFC 9580 §12.4), which removed the 9 "subkey, encryption" cells. Cells where only GopenPGP succeeds, or where Sequoia and GopenPGP disagree about what the right answer is, are not listed: for example unknown packets before the literal data (Sequoia refuses, GopenPGP and the draft's text accept; we refuse), and signatures with the old signature-before-literal layout (GopenPGP accepts; Sequoia and we refuse).

Reproduce: tools/oracles/interop-scoreboard.sh (once per OpenSSL), then interop-summary.py and interop-vs-sqop.py on results.json.

Left out of the scoreboard: GnuPG (no sop driver ships with it), RNP (none packaged), OpenPGP.js (the published sop driver, sop-openpgp-js 0.1.1, bundles OpenPGP.js 4 and does not start on the pinned Node.js 26; a driver over OpenPGP.js 6 is not written yet). They are covered by the two-way scripts and the corpus below.

Corpus

tools/oracles/corpus-*.sh and tools/oracles/pgpjs6/corpus.mjs generate keys, certificates, signatures and messages with throwaway keys; CorpusTests (set ONECRYPTO_OPENPGP_CORPUS) parses every file, checks that writing it back is byte-identical (from memory and through the stream reader), and compares fingerprints with each tool's own listing (primary key first, subkeys as a set).

Every file parses and rewrites identically. Two oracle behaviours are handled in the test, not the parser: gosop appends a line feed to a binary inline-signed message (trimmed, 4 files), and GnuPG writes LibrePGP version 5 keys (Ed448, Kyber) whose secret-key layout is read per the LibrePGP draft. Cleartext-signed messages are not binary OpenPGP data and are skipped.

Post-quantum samples come from Sequoia's mldsa65-ed25519 and mldsa87-ed448 suites (RFC 9980) and GnuPG 2.5's Kyber keys (LibrePGP); the pinned RNP and OpenPGP.js were not driven for post-quantum keys.