Results

What OneLLM measured about itself for the 1.0 release, and how. Every number below is OneLLM's own.

Run date: 2026-11-06. Commit: the 1.0.0 tag. Machine for the speed rows: AMD Ryzen 9 7950X3D, 16 physical and 32 logical cores, 125 GB, Linux Mint 22.3 (kernel 7.0.0-38), .NET SDK 10.0.403, runtime 10.0.14, BenchmarkDotNet 0.15.8, on an otherwise idle machine (1-minute load average between 1.1 and 8.4 during the runs).

Speed

Method: BenchmarkDotNet suites in bench/ against a local mock server that replays deterministic provider responses over loopback, so the numbers are client code and not a network or a vendor. The workloads, the exact settings and the way the gate runner records the machine and its load are in bench/README.md. Each row is the mean of the measured iterations; allocations are bytes per operation from the memory diagnoser.

A request row is a whole non-streamed turn: build the request from a history of the stated size, serialize it, send it over loopback and parse a one-line answer. A stream row reads a 2,000-event text stream (divide by 2,000 for the per-event cost); a first-event row stops at the first text event of a 50-event stream.

The agent loop

One run is eleven model turns (ten tool-calling turns and a final one) on the scripted provider, with the default runtime, an in-memory store and no record sinks: 78.2 us and 56.8 KB per run, 7.1 us and 5.2 KB per turn. That is above the fastest managed reference by a factor of 7.92, because every OneLLM run is a persisted instance with a snapshot, priced line items and events for each turn.

Floors, set at this first measurement and not to rise in later releases:

One observer hook on the message point adds about 0 us and 1.8 KB per run (78.6 us, 58.6 KB). Keeping the retained instance in memory costs 79.3 us per run. The same run with the file store, which writes the instance after every turn, takes 94.5 ms and 621 KB.

Other costs

Core ships no local tokenizer; its count comes from the provider's endpoint when one exists and otherwise from the calibrated estimate timed above.

Conformance

Each feature of each provider, on each deployment that has one, is replayed from recorded fixtures: the request bytes OneLLM sends and the result it parses are both checked. Streaming and whole-response calls must build equal messages. The core test suite (1,304 tests: 1,239 passed, 65 skipped for platform reasons, 0 failed) includes all of these.

TypeSafe Jev is covered by the decider checks below. The Bedrock request signing passes the published SigV4 test suite, and the Vertex service-account token passes the published JWT vector.

Parity and round trips

The reference corpus of definitions and instances is a private collection kept on the build machine; only counts are reported.

Loop behaviour, security and AOT

Live runs

A small smoke against each provider's real API, run once at release under hard spend caps (tools/live-smoke.sh): chat, streaming, a client tool round trip, structured output, a reasoning model, embeddings, token counting and, for OpenAI, a Decisions call. Then each of the eight example agents ran its representative task once on an OpenAI and a Gemini model (tools/examples-e2e.sh). The first OpenAI smoke passed 7 of 8 (the Decisions case used a wrong model name, fixed in the tool); the second passed 8 of 8.

Example agents, one run each. Outcome task_complete means the agent ended by calling task_complete; answered means it replied without calling it.

The Gemini cs-tool-writer row is its second attempt; the first stopped at its per-run spend cap after $0.0348, which is counted in the Gemini total above. A row is a full success when the agent calls task_complete, the files its task names exist, and its own checks passed (see the example agents); the answered rows are runs of the small OpenAI model that stopped without finishing the protocol.