# onellm docs, all pages (OneLLM 1.0.0) Every page of https://recursiveai.org/onellm/docs/ as markdown, getting started first. Index: https://recursiveai.org/onellm/llms.txt # Getting started with OneLLM Install OneLLM 1.0.0 for .NET 10, ask a model, get typed JSON back, stream, run a script, an agent with a C# tool and a decider, and see what each call cost. Every sample on this page was run on the 1.0.0 packages from NuGet, without an API key, and shows its real output. ## Install The packages target `net10.0`, so you need the .NET 10 SDK. ```shell dotnet new console -o OneLlmStart cd OneLlmStart dotnet add package RecursiveAI.OneLlm --version 1.0.0 ``` Seven packages are on NuGet at 1.0.0, all on one version: - **`RecursiveAI.OneLlm`**: the core. Provider clients, the model catalog and pricing, templating, the runtime, agents, tools, deciders, file and memory stores, JSONL and event sinks, OpenTelemetry. It brings in OneYaml and OneImage, and no native code. - **`RecursiveAI.OneLlm.Web`**: the `fetch_webpage_markdown` tool, on OneHtml. - **`RecursiveAI.OneLlm.JavaScript`**: JavaScript tools, hooks and decider logic in a sandbox, and `run_javascript`, on OneEcma. - **`RecursiveAI.OneLlm.Extensions.AI`**: OneLLM models as `IChatClient` and friends, and any `IChatClient` as a OneLLM provider. - **`RecursiveAI.OneLlm.EntityFrameworkCore`**: an instance store and a record sink on your `DbContext`. - **`RecursiveAI.OneLlm.OneLite`**: an instance store and a record sink on a OneLite file. - **`RecursiveAI.OneLlm.Cli`**: the `onellm` command-line tool. ## Keys OneLLM reads a provider’s key from the environment when the first call to that provider is made: | Provider | Variable | |---|---| | Anthropic | `ANTHROPIC_API_KEY` | | OpenAI | `OPENAI_API_KEY` | | Gemini | `GEMINI_API_KEY` | | TypeSafe Jev (deciders) | `TYPESAFE_API_KEY` | Without one, the call fails before anything is sent: ```text AuthException: anthropic: No API key. Set the environment variable ANTHROPIC_API_KEY. ``` Cloud deployments (Azure OpenAI / Foundry, Amazon Bedrock, Vertex AI) take explicit credentials; see [deployments](https://recursiveai.org/onellm/docs/deployments/). Coverage of providers, from the project’s [results](https://recursiveai.org/onellm/docs/results/): every provider and deployment is replayed from recorded fixtures; a live smoke at release passed against OpenAI (8 of 8) and Gemini (7 of 7); Anthropic, Azure OpenAI / Foundry, Amazon Bedrock and Vertex AI were **not run live**. ## Run it without a key To try every sample below with no key and no spend, start `Program.cs` with this block. It registers the library’s own test provider, `OneLlm.Testing.ScriptedProvider`, under the real provider names, so model names resolve and the shipped catalog prices each call as it would a real one. Each sample then queues the answer the scripted model gives. With a real key, delete this block and the `Enqueue` lines; the rest is unchanged. ```csharp using System.Text.Json.Serialization; using OneLlm; using OneLlm.Agents; using OneLlm.Deciders; using OneLlm.Messages; using OneLlm.Testing; using OneLlm.Tools; var anthropic = new ScriptedProvider("anthropic"); var openai = new ScriptedProvider("openai"); var noKey = new Runtime(); noKey.Providers.Set("anthropic", anthropic); noKey.Providers.Set("openai", openai); noKey.Providers.Set("typesafe", new ScriptedDecisionProvider("typesafe")); noKey.Tools.Add(); Llm.Default = noKey; ``` ## A first call A model is a catalog name or alias, such as `claude-sonnet`, `gpt-6-luna` or `gemini-flash`; `provider:model` says which provider serves it (`openai:gpt-6-luna`, `ollama:llama3`). `GenerateAsync` returns the same call with its usage and cost: ```csharp anthropic.Enqueue(ScriptStep.TextWithUsage("A monad is a type that wraps a value and chains computations on it.", 14, 15)); string answer = await Llm.AskAsync("claude-sonnet", "In one sentence, what is a monad?"); Console.WriteLine(answer); openai.Enqueue(ScriptStep.TextWithUsage("2, 3 and 5.", 13, 7)); AskResult r = await Llm.GenerateAsync("gpt-6-luna", "Name three prime numbers.", []); Console.WriteLine($"{r.Text} [{r.Usage?.InputTokens} in, {r.Usage?.OutputTokens} out, {r.Cost.Total}]"); foreach (CostLine line in r.Cost.Lines) Console.WriteLine($" {line.Counter}: {line.Quantity} x {line.UnitPrice} per {line.Per} = {line.Amount}"); ``` ```text A monad is a type that wraps a value and chains computations on it. 2, 3 and 5. [13 in, 7 out, 0.0000048] inputTokens: 13 x 0.1 per 1000000 = 0.0000013 outputTokens: 7 x 0.5 per 1000000 = 0.0000035 ``` `Cost` is a list of line items plus a `decimal` total, priced from the catalog (here `gpt-6-luna` at $0.10 and $0.50 per million tokens, as of 2026-11-05). A model with no price in the catalog gives zero cost and a warning, never a guess. To see a model’s values and where each came from, run `onellm catalog ` (below). ## Structured output The schema comes from your type and is sent through the provider’s native structured output. Passing the source-generated type info keeps the call trim and NativeAOT safe. ```csharp openai.Enqueue(ScriptStep.TextWithUsage("{\"customer\":\"ACME Ltd\",\"total\":120.50}", 60, 12)); Invoice invoice = await Llm.AskAsync("gpt-6-luna", "ACME Ltd billed 120.50 for consulting.", AppJson.Default.Invoice); Console.WriteLine(invoice); ``` ```text Invoice { Customer = ACME Ltd, Total = 120.50 } ``` `Invoice` and `AppJson` go at the end of `Program.cs`, with the tool class used further down: ```csharp public sealed record Invoice(string Customer, decimal Total); [JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase)] [JsonSerializable(typeof(Invoice))] public sealed partial class AppJson : JsonSerializerContext; public sealed partial class DocketTools { /// Looks up a docket entry. /// Case number, for example 24-1234. [Tool] public static string GetDocket(string caseNo) => "Docket " + caseNo + ": open"; } ``` ## Streaming Streamed events build the same result a whole call returns. ```csharp anthropic.Enqueue(ScriptStep.Text("Green lights at midnight, ", "the queue drains one job at a time, ", "dawn finds the build red.")); StreamingAnswer stream = Llm.StreamAsync("claude-haiku", "Write a haiku about build servers."); await foreach (string piece in stream.TextAsync()) Console.Write(piece); AskResult done = await stream.Result; Console.WriteLine(); Console.WriteLine($"result text matches: {done.Text == "Green lights at midnight, the queue drains one job at a time, dawn finds the build red."}"); ``` ```text Green lights at midnight, the queue drains one job at a time, dawn finds the build red. result text matches: True ``` ## A script A `.flow` file is a script: its prompts are templates in Go’s `text/template` language, rendered in order, one model call per user prompt, no tools. Save this as `prompts/commit-message.flow`: ```yaml model: gpt-6-luna systemPrompt: You write git commit messages. initialPrompts: - user: - text: "Summarize this change in one sentence: {{ .inputs.diff }}" - user: - text: "Now write the commit message from your summary: {{ (index .turns 0).text }}" ``` ```csharp string diff = "- retries = 1\n+ retries = 3"; openai.Enqueue(ScriptStep.TextWithUsage("Raises the retry count from 1 to 3.", 40, 10), ScriptStep.TextWithUsage("Raise retries from 1 to 3\n\nTransient failures now get two more attempts.", 70, 16)); ScriptResult script = await Llm.RunScriptAsync("prompts/commit-message.flow", new Dictionary { ["diff"] = diff }); Console.WriteLine(script.Text); Console.WriteLine($"calls: {script.Turns.Count}, cost: {script.Cost.Total}"); ``` ```text Raise retries from 1 to 3 Transient failures now get two more attempts. calls: 2, cost: 0.0000240 ``` See [templating](https://recursiveai.org/onellm/docs/templating/) for the language. ## An agent with a C# tool A tool is a method marked `[Tool]` on a partial class (`DocketTools` above). The generator writes the JSON schema from the XML comments and an invoker that uses no reflection. Save the agent as `agents/clerk.yaml`: ```yaml name: clerk model: claude-sonnet systemPrompt: You look up court dockets. Call task_complete with your answer. inputs: caseNo: "" initialPrompts: - user: - text: "Find docket {{ .inputs.caseNo }} and report its status." references: - tools: [get_docket, task_complete] ``` The scripted model calls `get_docket`, then `task_complete`: ```csharp anthropic.Enqueue( new ScriptStep { ToolCalls = [new ScriptedToolCall("c1", "get_docket", "{\"caseNo\":\"24-1234\"}")], Usage = new Usage { InputTokens = 300, OutputTokens = 20 } }, new ScriptStep { ToolCalls = [new ScriptedToolCall("c2", "task_complete", "{\"result\":\"Docket 24-1234 is open.\",\"summary\":\"Looked up the docket.\"}")], Usage = new Usage { InputTokens = 340, OutputTokens = 25 } }); AgentResult agent = await Llm.RunAgentAsync("agents/clerk.yaml", new Dictionary { ["caseNo"] = "24-1234" }); Console.WriteLine($"{agent.StopReason}: {agent.Text} ({agent.Cost.Total})"); ``` ```text Completed: Docket 24-1234 is open. (0.001730) ``` Two turns of `claude-sonnet` (640 tokens in, 45 out) at the catalog’s $2 and $10 per million. The loop ends when `task_complete` runs, the model answers with no tool call, a call needs approval, or `maxIterations` is reached. `Llm.StartAgentAsync` returns a run you can follow, steer and approve; approvals are saved in the instance. See [agents](https://recursiveai.org/onellm/docs/agents/). ## A decider A decider asks typed questions of a model and returns the answers. Save this as `deciders/triage-ticket/decider.yaml`: ```yaml model: jev-latest state: title: "{{ .inputs.title }}" questions: topic: type: choice instructions: Which team should handle the ticket? criteria: billing: Charges and refunds orders: Delivery and returns angry: type: noul instructions: Is the customer angry? ``` ```csharp DecisionResult verdict = await Llm.DecideAsync("deciders/triage-ticket", new Dictionary { ["title"] = "Refund please" }); Console.WriteLine(verdict.Answers.GetRawText()); ``` ```text {"topic":{"type":"choice","choice":"billing","confidence":0.5,"probabilities":{}},"angry":{"type":"noul","noul":0.5}} ``` These are the scripted engine’s placeholder answers (the first choice, 0.5); a real engine fills in its own. The engine comes from the model’s catalog capabilities: TypeSafe Jev, OpenAI Decisions or any chat model. See [deciders](https://recursiveai.org/onellm/docs/deciders/). ## Grants and spend caps A definition names tools; the host decides what they may touch. The default runtime allows files under the working directory, with the shell and the network off. To grant more, build a `Runtime`. The web tool here is in its own package: ```shell dotnet add package RecursiveAI.OneLlm.Web --version 1.0.0 ``` ```csharp var runtime = new Runtime { Capabilities = Capabilities.Files("work") + Capabilities.ReadOnlyFiles("reference") + Capabilities.ShellCommands("git status") + Capabilities.Network("api.example.com"), Store = new OneLlm.Instances.FileInstanceStore("runs"), SpendCap = 5m, // dollars, for the whole runtime InstanceSpendCap = 1m, }; runtime.Tools.Add(); runtime.Tools.Add(new OneLlm.Web.FetchWebpageMarkdownTool()); // RecursiveAI.OneLlm.Web; needs the Network grant ``` Set it as `Llm.Default` to use it from the statics. A call that could pass a cap throws `BudgetExceededException` before anything is sent. See [capabilities](https://recursiveai.org/onellm/docs/capabilities/) and [the runtime](https://recursiveai.org/onellm/docs/happy-path/). ## From Microsoft.Extensions.AI With the bridge package, a OneLLM model is an `IChatClient`, with its cost in `AdditionalProperties`: ```shell dotnet add package RecursiveAI.OneLlm.Extensions.AI --version 1.0.0 ``` ```csharp anthropic.Enqueue(ScriptStep.TextWithUsage("Hello from the bridge.", 9, 5)); Microsoft.Extensions.AI.IChatClient client = OneLlm.Extensions.AI.RuntimeExtensions.AsChatClient(noKey, "claude-sonnet"); var response = await client.GetResponseAsync([new Microsoft.Extensions.AI.ChatMessage(Microsoft.Extensions.AI.ChatRole.User, "Say hello.")]); Console.WriteLine($"{response.Text} {response.AdditionalProperties?["onellm.cost"]}"); ``` ```text Hello from the bridge. $0.000068 ``` With `using OneLlm.Extensions.AI;` the same is `runtime.AsChatClient("claude-sonnet")`. See [the bridge](https://recursiveai.org/onellm/docs/extensions-ai/), and [coming from another library](https://recursiveai.org/onellm/docs/reads-like/) for call-for-call tables. ## The onellm tool ```shell dotnet tool install --global RecursiveAI.OneLlm.Cli --version 1.0.0 onellm catalog claude-sonnet ``` ```text model: anthropic:claude-sonnet-5-5 provider: anthropic name: "claude-sonnet-5-5" [shipped] displayName: "Claude Sonnet 5.5" [shipped] contextWindow: 1000000 [shipped] … pricing.inputPerMillion: 2 [shipped] pricing.outputPerMillion: 10 [shipped] pricing.asOf: "2026-11-05" [shipped] ``` `onellm run` runs an agent, a `.flow` or a decider. Here the flow above, with its model changed to `ollama:llama3`, runs against a local OpenAI-compatible server on Ollama’s port (for this page, a stand-in that answers with fixed text). The result goes to standard output; the cost line to standard error: ```shell onellm run commit-message.flow --input diff=$'- retries = 1\n+ retries = 3' onellm cost ``` ```text Raise retries from 1 to 3 Transient failures now get two more attempts before the job fails. script i-8d48971163424f3ebc6fa8efe5c2a255 · 2 turns · cost $0.000000 llama3 2 calls 80 in 24 out $0.000000 1 instance: $0.000000 ``` Presets for servers on your own machine are priced at zero. The run is saved as a readable YAML instance under `.onellm/instances/`. Without a key for a hosted model the tool exits with code 3 and names the variable to set. See [the onellm tool](https://recursiveai.org/onellm/docs/cli/) and [the example agents](https://recursiveai.org/onellm/docs/examples/) it embeds. ## Next - [The happy path and the runtime](https://recursiveai.org/onellm/docs/happy-path/): the statics, instances, spend caps, testing with the scripted provider. - [Agents](https://recursiveai.org/onellm/docs/agents/): approvals, steering, sub-agents, monitors and hooks. - [Pricing and token counting](https://recursiveai.org/onellm/docs/pricing-and-counting/): catalog layers and adding a model or a price. - [Results](https://recursiveai.org/onellm/docs/results/): speed, conformance, parity and live runs, with method. # OneLLM **LLM inference and agent runtime for .NET.** One message model and one set of calls over Anthropic, OpenAI and Gemini (first-party and on Azure, Bedrock and Vertex) and OpenAI-compatible servers. A layered model catalog prices every call in `decimal` line items. Agents, scripts and deciders are files you can read, run, persist, fork and resume. The core depends on the base class library, OneYaml and OneImage only, and is trim and NativeAOT safe on its main paths. NuGet: `RecursiveAI.OneLlm` (namespace `OneLlm`). Requires .NET 10. > OneLLM is unaffiliated with the `onellm` package on PyPI and with the OneLLM research model. ## What it is - **Provider clients** for each provider's inference surface: chat, streaming, client and hosted tools, structured output, reasoning replay, caching, token counting, batch, embeddings, plus image generation, speech and moderation where the provider has them. - **One runtime, three ways to run:** simple calls (`Ask`, `Stream`, `Chat`), scripts (`.flow` files: prompts in order, one call each) and agents (the tool loop with approvals, steering, sub-agents, monitors and context strategies). Every run is an instance saved in a store you choose. - **Deciders:** typed choice, score and yes/no judgments on TypeSafe Jev, OpenAI Decisions or any chat model. - **Cost on everything:** pricing is metadata (catalog files you can override), and every result, tool call and instance subtree carries a line-item `Cost`. - **Safe by default:** loading a definition never grants anything. The host grants file roots, shell and network; the default is files under the working directory only. - **Records and telemetry:** instance stores (file, memory, OneLite, Entity Framework Core), record sinks (JSONL, events, OneLite, EF) and OpenTelemetry GenAI spans and metrics. ## Install ``` dotnet add package RecursiveAI.OneLlm ``` Set the key of the provider you use: `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` or `GEMINI_API_KEY`. See [getting started](https://recursiveai.org/onellm/docs/getting-started/). ## The happy path ```csharp using OneLlm; string s = await Llm.AskAsync("claude-sonnet", "Summarize this", Asset.File("brief.pdf")); Invoice inv = await Llm.AskAsync("gpt-6-luna", "Extract the invoice", Asset.File("scan.png"), AppJson.Default.Invoice); await foreach (string piece in Llm.StreamTextAsync("gemini-flash", "Write a haiku")) Console.Write(piece); float[] v = await Llm.EmbedAsync("embedding-small", "reset my password"); using ChatSession chat = Llm.Chat("claude-sonnet"); await chat.AskAsync("Name a prime."); await chat.AskAsync("And the next one?"); AskResult r = await Llm.GenerateAsync("gpt-6-luna", "Name three prime numbers.", []); Console.WriteLine($"{r.Text} [{r.Usage?.InputTokens} in, {r.Usage?.OutputTokens} out, {r.Cost.Total}]"); ``` `Invoice` is your record and `AppJson` a `JsonSerializerContext` that lists it, which keeps structured output trim and NativeAOT safe. Models are catalog names or aliases; `provider:model` says which provider. Agents, scripts and deciders: ```csharp ScriptResult script = await Llm.RunScriptAsync("prompts/commit-message.flow", new Dictionary { ["diff"] = diff }); AgentResult agent = await Llm.RunAgentAsync("agents/clerk.yaml", new Dictionary { ["caseNo"] = "24-1234" }); DecisionResult d = await Llm.DecideAsync("deciders/triage-ticket", new Dictionary { ["title"] = "Refund please" }); ``` Under the statics is `Runtime`, which owns the catalog, providers, stores, sinks, tools, hooks and grants. See [the happy path and the runtime](https://recursiveai.org/onellm/docs/happy-path/). ## Providers and deployments Status is what the repository tests today: **recorded fixtures** replay request bytes and parsed results for every row; a one-time live smoke at release passed against OpenAI and Gemini, and Anthropic, Azure OpenAI / Foundry, Amazon Bedrock and Vertex AI were not run live (see [results](https://recursiveai.org/onellm/docs/results/)). | Provider | Deployments | Covered | |---|---|---| | Anthropic (Messages) | first-party, Amazon Bedrock, Vertex AI | chat, streaming, tools, hosted tools, structured output, thinking replay, caching, token counting, message batches | | OpenAI (Responses; Chat Completions for compatible servers) | first-party, Azure OpenAI / Foundry | chat, streaming, tools, hosted tools, structured output, reasoning items, caching, batch, embeddings, images, speech, moderation, Decisions | | Gemini (`generateContent`) | Gemini API, Vertex AI | chat, streaming, tools, hosted tools, structured output, thought signatures, caching, token counting, batch, embeddings, images, speech | | OpenAI-compatible | any base URL; presets for Ollama, vLLM, LM Studio, OpenRouter, Groq, DeepSeek, xAI, Mistral, Together | what the server implements | | TypeSafe Jev | first-party | decisions | See [deployments](https://recursiveai.org/onellm/docs/deployments/), [OpenAI](https://recursiveai.org/onellm/docs/providers-openai/), [Gemini](https://recursiveai.org/onellm/docs/gemini/) and [media](https://recursiveai.org/onellm/docs/media/). ## Packages All packages share one version. | Package | Adds | Depends on | |---|---|---| | `RecursiveAI.OneLlm` | message model, catalog and pricing, providers and deployments, templating, runtime, agents, tools and generators, deciders, file and memory stores, JSONL and event sinks, OpenTelemetry | BCL, OneYaml, OneImage | | `RecursiveAI.OneLlm.Web` | `fetch_webpage_markdown` | core, OneHtml | | `RecursiveAI.OneLlm.JavaScript` | JavaScript tools, hooks and decider logic in a sandbox, and `run_javascript` | core, OneEcma | | `RecursiveAI.OneLlm.Extensions.AI` | OneLLM models as `IChatClient` and friends; any `IChatClient` as a provider; `AIFunction` and `ITool` both ways | core, Microsoft.Extensions.AI.Abstractions | | `RecursiveAI.OneLlm.EntityFrameworkCore` | instance store and record sink on your `DbContext` | core, EF Core | | `RecursiveAI.OneLlm.OneLite` | instance store and record sink on a OneLite file | core, OneLite | | `RecursiveAI.OneLlm.Cli` | the `onellm` command-line tool (`dotnet tool install -g RecursiveAI.OneLlm.Cli`) with eight embedded example agents | core, `.Web`, `.JavaScript` | The source generators for `[Tool]` and `[Decider]` ship inside the core package. ## Documentation | | | |---|---| | [Getting started](https://recursiveai.org/onellm/docs/getting-started/) | keys, model names, first call, structured output, an agent with a C# tool, a decider, grants | | [The happy path and the runtime](https://recursiveai.org/onellm/docs/happy-path/) | the statics, instances, spend caps, scripted runs | | [Agents](https://recursiveai.org/onellm/docs/agents/) | the loop, approvals, steering, sub-agents, monitors, hooks | | [Capabilities and built-in tools](https://recursiveai.org/onellm/docs/capabilities/) | grants and the built-in tools | | [Deciders](https://recursiveai.org/onellm/docs/deciders/) | the three engines and the run APIs | | [Pricing and counting](https://recursiveai.org/onellm/docs/pricing-and-counting/) | catalog layers, line-item cost, token counts | | [Templating](https://recursiveai.org/onellm/docs/templating/) | the prompt template language | | [Stores and sinks](https://recursiveai.org/onellm/docs/stores/) | where instances and records go | | [Telemetry](https://recursiveai.org/onellm/docs/telemetry/) | OpenTelemetry names | | [Errors and retries](https://recursiveai.org/onellm/docs/errors-and-retries/) | typed errors, `Retry-After` | | [Microsoft.Extensions.AI bridge](https://recursiveai.org/onellm/docs/extensions-ai/) | interop both ways | | [Reads like](https://recursiveai.org/onellm/docs/reads-like/) | call-for-call tables from Microsoft.Extensions.AI, Microsoft Agent Framework and the vendor SDKs | | [JavaScript tools and hooks](https://recursiveai.org/onellm/docs/javascript/) | tool files, hooks, decider logic, `run_javascript` | | [The `onellm` command line](https://recursiveai.org/onellm/docs/cli/) | verbs, grants, exit codes, the example agents | | [Results](https://recursiveai.org/onellm/docs/results/) | speed, conformance, parity, AOT and live-run results, with method | | [`spec/`](https://github.com/recursive-ai-oss/onellm-csharp/tree/38ccb22024da28a34dac5d87d1ff83fae2656dee/spec) | the file formats: definitions, instances, catalog, messages, records | ## Build ``` dotnet build OneLlm.slnx dotnet test --solution OneLlm.slnx ``` Build output goes to the directory in `ONELLM_ARTIFACTS`, or to `artifacts/` in the repository. ## Licence `MIT OR Apache-2.0`, Copyright (c) Recursive AI, LLC. See `LICENSE-MIT` and `LICENSE-APACHE`. # The happy path and the runtime beneath it OneLLM has two ways in. The statics on `Llm` are verbs that take a model name and what to ask. Underneath is `Runtime`, which the statics use through `Llm.Default`; build your own when you want to choose where instances are kept, add record sinks, set spend caps or register a provider. ```csharp string s = await Llm.AskAsync("claude-sonnet", "Summarize this", Asset.File("brief.pdf")); Invoice inv = await Llm.AskAsync("gpt-5.5", "Extract the invoice", Asset.File("scan.png"), AppJson.Default.Invoice); await foreach (string piece in Llm.StreamTextAsync("gemini-3-pro", "Write a haiku")) Console.Write(piece); float[] v = await Llm.EmbedAsync("text-embedding-4", "reset my password"); using var chat = Llm.Chat("claude-sonnet"); await chat.AskAsync("…"); await chat.AskAsync("and then?"); ScriptResult r = await Llm.RunScriptAsync("prompts/commit-message.flow", new Dictionary { ["diff"] = diff }); ``` ## Models A model is a catalog name or alias (`claude-sonnet`), or `provider:model` (`openai:gpt-5.5`, or a preset such as `ollama:llama3`) to say which provider. An unknown name throws `ModelNotFoundException`; a name that two providers claim throws it too, with the candidates listed. The catalog is the shipped files, then the files named by `ONELLM_CATALOG`, then code overrides. ## The statics | Call | Returns | Notes | |---|---|---| | `AskAsync(model, prompt)` / `AskAsync(model, prompt, asset)` | `string` | The assistant's text. | | `AskAsync(model, prompt, typeInfo)` | `T` | Native structured output; `typeInfo` from a source-generated `JsonSerializerContext`. Without it, `AskAsync(model, prompt)` finds the type by reflection and is not trim or AOT safe. `T` must be a JSON object type. | | `GenerateAsync(model, prompt, attachments)` / `GenerateAsync(…, typeInfo)` | `AskResult` / `AskResult` | Text or value, plus `Usage`, `Cost` (line items and total), `Warnings`, the saved `Instance`, `Finish`. | | `StreamAsync(model, prompt)` | `StreamingAnswer` | Events as they arrive; after the loop, `await stream.Result`. | | `StreamTextAsync(model, prompt)` | `IAsyncEnumerable` | Text pieces only. | | `EmbedAsync(model, text)` / `EmbedAsync(model, texts)` | `float[]` / `float[][]` | | | `Chat(model, system)` | `ChatSession` | `AskAsync`, `SendAsync`, `StreamAsync`; one instance for the whole conversation. | | `RunScriptAsync(path, inputs)` | `ScriptResult` | `.flow` or agent definition run as a script. | Every call takes a `CancellationToken`. Assets come from `Asset.File(path)` and `Asset.FromBytes(bytes, contentType)`. Inputs to a script are a dictionary (or a value with a `JsonTypeInfo`) on the AOT-safe path. Passing an anonymous object works by reflection and is marked `RequiresUnreferencedCode`. ## Every call is an instance A call creates an instance, runs it, and saves it in the runtime's store (in memory unless `ONELLM_INSTANCES` names a directory, or you set `Runtime.Store`). The instance holds the messages with their usage and cost, so a chat can be listed, forked and resumed. Records go to the runtime's sinks (`ONELLM_RECORDS` adds a JSONL directory) and each model call is a trace span and metrics (see [telemetry](https://recursiveai.org/onellm/docs/telemetry/)). Embeddings are stateless: they make no instance, but are recorded and priced. The instance states a run ends in: `stopped` (an ask or a script finished), `waiting` (a chat between turns), `error`. A cancelled run ends `stopped` with `cancelled: true` in its extension data, keeps what had arrived (a streamed partial answer is saved as the last assistant message), and throws `RunCancelledException`, an `OperationCanceledException` that carries the instance. ## Cost and spend caps Usage is priced from the catalog (see [pricing](https://recursiveai.org/onellm/docs/pricing-and-counting/)); an unpriced model gives `Cost.Zero` and a `pricing` warning. `Runtime.SpendCap` (whole runtime), `Runtime.InstanceSpendCap` (one instance, counting its earlier turns) and `Runtime.MaxCostPerCall` are checked before a call against its worst case (local token estimate plus the full output allowance) and after against the real cost. A call that could pass a cap throws `BudgetExceededException` before anything is sent. ## Scripted runs `RunScriptAsync` renders the definition's `systemPrompt` and each of its `initialPrompts` as Go-template text (see [templating](https://recursiveai.org/onellm/docs/templating/)) and sends the rendered system prompt with each call. Each user prompt gets one model call, in order, with everything before it as history; no tools are offered, and tool references and tool calls are ignored with a warning. Assistant prompts are added to the history without a call. The template scope: | Name | Holds | |---|---| | `.inputs` | The definition's `inputs` defaults, overridden by the caller's values. | | `.system` | `os`, `arch`, `hostname`, `username`, `homeDir`, `tempDir`, `workingDir`. | | `.time` | `now` (RFC 3339), `today`, `timezone`, from the runtime's clock. | | `.env` | Empty, except for the names in `Runtime.AllowedEnvironment`. A prompt cannot read a credential the host did not list. | | `.git` | Empty fields; the host fills it through `Runtime.ScopeValues`. | | `.swarm`, `.caller`, `.invocationMode` | Extension data, the calling instance, and the definition's mode (default `autonomous`). | | `.turns` | The earlier answers: `.turns[0].text`, and `.turns[0].json` when the text is JSON. | `missingKey: error` in the definition makes a missing key a render error; otherwise it prints ``. `embed` reads files beside the definition (never outside its directory) and then `Runtime.EmbedSources`. A prompt's `when` is a template: the prompt runs unless it renders to empty, `false`, `0`, `no` or `off` (a `when` that fails to render counts as true). The rendered system prompt is stored as `metadata.renderedSystemPrompt` in the instance. A definition with an `outputSchema` (JSON Schema, an extra field of the definition) asks for that shape on its last user prompt; `ScriptResult.Output` holds the JSON. ## Providers `Runtime.Providers` maps a catalog provider name (or preset name, or preset protocol) to a provider, created on first use from the environment variable the catalog names (`apiKeyEnvVar`). A missing variable throws `AuthException` naming it; no registered provider throws `CapabilityMissingException` naming what to register. `ProviderRegistry.Set(name, provider)` plugs in a ready provider. ## Testing code that uses OneLLM `OneLlm.Testing.ScriptedProvider` replays a script of answers (text, tool calls, usage, failures, pauses) and records the requests it received. Streaming and whole answers go through the same events, so they build equal messages. ```csharp var provider = new ScriptedProvider().EnqueueText("hello"); var runtime = new Runtime { Catalog = myCatalog, Providers = new ProviderRegistry() }; runtime.Providers.Set("scripted", provider); Llm.Default = runtime; // for the statics; tests that do this must not run in parallel with others that use them ``` # Reads like If you know another .NET LLM library, these tables show where the same job lives in OneLLM. Each row names a call or concept **as its own documentation presents it**, and the OneLLM call that does the same job. They are a map, not a compatibility layer: OneLLM does not copy the other libraries' shapes, and the tables make no claim about how the libraries compare. Where a library documents a feature OneLLM does not have, the row says so. The sources are public documentation pages only; they are listed at the end. In the examples, `rt` is a `Runtime` (`Llm.Default` for the statics), and `model` is a catalog name or alias. ## Microsoft.Extensions.AI Using OneLLM *as* a `Microsoft.Extensions.AI` client, or putting a client behind OneLLM, is the job of `RecursiveAI.OneLlm.Extensions.AI`; see [the bridge](https://recursiveai.org/onellm/docs/extensions-ai/). The table below is for code that would call OneLLM directly instead. | Microsoft.Extensions.AI documents | OneLLM | |---|---| | `client.GetResponseAsync("What is AI?")` | `Llm.AskAsync(model, "What is AI?")` for the text; `Llm.GenerateAsync(model, "What is AI?", [])` for usage and cost | | `GetResponseAsync(messages)` with a `List` you keep and extend (`history.AddMessages(response)`) | `Llm.Chat(model)` returns a `ChatSession` that keeps the conversation as one saved instance: `await chat.AskAsync(...)` per turn. Or build a `ChatRequest` and call `IChatProvider.SendAsync` yourself. | | `GetStreamingResponseAsync("...")`, iterating `ChatResponseUpdate` | `Llm.StreamTextAsync(model, "...")` for text pieces; `Llm.StreamAsync(model, "...")` for all stream events, then `await stream.Result` | | `ToChatResponse(updates)` to compose a streamed response | not needed: the stream builds the same `Message` a whole call builds, available as `stream.Result` | | `ChatOptions.Temperature`, `MaxOutputTokens` | `AskRequest.Temperature`, `AskRequest.MaxOutputTokens` through `Runtime.AskAsync`; `temperature` and `maxTokens` in a definition | | `ChatOptions.ModelId` | the `model` argument; `provider:model` picks the provider | | `ChatOptions.ConversationId` (a service-held conversation) | not used: OneLLM keeps the conversation in the instance and sends the history. `Llm.Chat` gives the same continuity. | | `ChatOptions.AdditionalProperties` / `RawRepresentationFactory` | `AskRequest.ExtraParameters` (merged into the request body) | | `AIFunctionFactory.Create(method)` and `ChatOptions.Tools` | a `[Tool]` method on a partial class, registered with `rt.Tools.Add()` and named in a definition's `references`. `Tool.Create(delegate)` is the reflection quick path (not AOT safe). | | `UseFunctionInvocation()` (run the tools the model asks for, loop) | an agent: `Llm.RunAgentAsync(path, inputs)`. The loop runs calls in parallel unless a tool says otherwise, returns results in call order, and keeps approvals, steering and a spend cap. | | `UseDistributedCache()` (reuse a response for an identical history) | the opt-in result cache, `Runtime.Cache`, off by default; provider prompt caching is separate and described in [pricing and counting](https://recursiveai.org/onellm/docs/pricing-and-counting/) | | `UseOpenTelemetry()` | always on: `ActivitySource` and `Meter` named `OneLlm`, GenAI names; subscribe with `AddSource("OneLlm")`. See [telemetry](https://recursiveai.org/onellm/docs/telemetry/). | | `UseLogging()` | record sinks (`JsonlRecordSink`, `EventRecordSink`) and opt-in scrubbed raw turn logs; see [stores](https://recursiveai.org/onellm/docs/stores/) | | `DelegatingChatClient` middleware | hooks (`Runtime.Hooks`): before and after inference, before and after tool calls, on message, on cost; see [agents](https://recursiveai.org/onellm/docs/agents/) | | `IEmbeddingGenerator>` | `Llm.EmbedAsync(model, text)` / `EmbedAsync(model, texts)`, or `IEmbeddingProvider.EmbedAsync` | | `IImageGenerator` (experimental) | `IImageProvider.GenerateImageAsync`, the `generate_image` built-in tool; see [media](https://recursiveai.org/onellm/docs/media/) | | `ChatResponseFormat` / structured output | `Llm.AskAsync(model, prompt, typeInfo)`, native structured output on each provider | | `UsageDetails` on a response | `AskResult.Usage` and `AskResult.Cost` (line items and a `decimal` total) | | Chat reduction (`MessageCountingChatReducer`, `SummarizingChatReducer`, experimental) | context strategies on a definition (`clearToolResults`, `compact`, `providerNative`); see [agents](https://recursiveai.org/onellm/docs/agents/) | | Chat routing and failover (experimental) | not provided as a type; `ProviderRegistry` lets you register any provider under a name, and an `Extensions.AI` `IChatClient` can be one | ## Microsoft Agent Framework | Agent Framework documents | OneLLM | |---|---| | Create an `AIAgent` from a client with a model and `instructions`; `agent.RunAsync("...")` | a definition (`model`, `systemPrompt`) run by `Llm.RunAgentAsync(path, inputs)`, which returns an `AgentResult`; or `rt.StartAgentAsync` for a live `AgentRun` | | Agent instructions as a string | `systemPrompt`, a template rendered when the run starts (see [templating](https://recursiveai.org/onellm/docs/templating/)) | | Function tools (`AIFunction`s given to the agent) | `[Tool]` methods; `references: - tools: [name, ...]` in the definition; MCP tool sources are not in this release | | Multi-turn conversations with an agent session | an instance: the conversation is saved after every step; `Llm.Chat` for plain chat, and `rt.GetAgentAsync(instanceId)` to pick an agent run up again, including after a restart | | Persisting and resuming conversations | `IInstanceStore` (file, memory, OneLite, EF Core); instance files are readable YAML; fork and duplicate with `rt.ForkAsync` / `rt.DuplicateAsync` | | Structured output from an agent | `outputSchema` on a definition with a script run (`ScriptResult.Output`); `AskAsync` for a single call | | Middleware that intercepts agent actions | hooks; interceptors fail closed, observers fail open, and every intervention is recorded | | Harness agent: planning and todo tracking | the `todo` built-in tool | | Harness agent: file access and memory | the file built-ins under granted roots; the `memory` built-in | | Harness agent: context compaction | context strategies, including a scripted `compact` definition you can override | | Harness agent: don't-ask-again tool approval | a tool marked `RequiresApproval`, or a hook returning `RequireApproval`; approvals persist across restarts. There is no "don't ask again" memory; approve per call. | | Harness agent: observability | OpenTelemetry spans and metrics, record sinks, per-instance and per-subtree cost | | Multi-agent patterns | sub-agents (`run_agent_async`, `await_agent`, `message_agent`) in one `Runtime`, each child bounded by its parent's grants | | Workflows (functional and graph-based) | not provided. Scripts (`.flow`) run prompts in sequence; graph workflows are a later item. | | Hosted MCP tools, A2A | not in this release | ## OpenAI .NET library The OpenAI documentation page shows a `ResponsesClient` created with a key, `CreateResponseAsync(model, input)` and `GetOutputText()`. The rest of the table follows the API guides the same documentation set links to. | OpenAI documents | OneLLM | |---|---| | `new ResponsesClient(key)`; `CreateResponseAsync("gpt-...", "Say 'this is a test.'")`, then `GetOutputText()` | `await Llm.AskAsync("gpt-...", "Say 'this is a test.'")`; the key is read from `OPENAI_API_KEY` | | Responses API with conversation state | the `OpenAIProvider` speaks Responses; the history is sent from the instance | | Streaming responses | `Llm.StreamTextAsync` / `Llm.StreamAsync` | | Function calling | `[Tool]` methods in an agent; or `ChatRequest.Tools` and read `ToolCalls` yourself at the low level | | Structured output | `Llm.AskAsync` | | Reasoning models, reasoning effort | `thinking: { effort: ... }` in a definition; `AskRequest.Reasoning`; reasoning items are kept and replayed | | Embeddings | `Llm.EmbedAsync` | | Moderation | `IModerationProvider.ModerateAsync`, the `moderate` tool | | Image generation | `IImageProvider.GenerateImageAsync`, the `generate_image` tool | | Text to speech, speech to text | `ITextToSpeechProvider.SynthesizeAsync`, `ISpeechToTextProvider.TranscribeAsync` | | Batch | `IBatchProvider.SubmitBatchAsync`, `GetBatchAsync`, `GetBatchResultsAsync` | | Counting tokens | `rt.TokenCounter.CountAsync(request, model, provider)`; every count says which tier produced it | | Decisions API | `Llm.DecideAsync` with a decider on `openai-decisions` models; see [deciders](https://recursiveai.org/onellm/docs/deciders/) | | Azure OpenAI libraries | `AzureOpenAIEndpoint` on `OpenAIProvider`; see [deployments](https://recursiveai.org/onellm/docs/deployments/) | | Prompt caching | `cache` on a message; cached tokens are priced at their own rate | ## Anthropic C# SDK The Anthropic documentation shows `new AnthropicClient()` (key from `ANTHROPIC_API_KEY`), a `MessageCreateParams` with `MaxTokens`, `Messages` and `Model`, `client.Messages.Create(parameters)` returning content blocks, `CreateStreaming`, `WithOptions` for per-call settings, `AsIChatClient`, and separate packages for Bedrock and Vertex. | Anthropic documents | OneLLM | |---|---| | `client.Messages.Create(new() { Model, MaxTokens, Messages = [...] })`, then reading text blocks | `Llm.AskAsync("claude-sonnet", "Hello, Claude")`; `max_tokens` is set from the catalog (`AskRequest.MaxOutputTokens` to change it) | | `Messages.CreateStreaming(parameters)` | `Llm.StreamTextAsync("claude-sonnet", ...)` or `Llm.StreamAsync` | | `AnthropicClient { ApiKey = ... }` | the key comes from `ANTHROPIC_API_KEY`; an explicit key goes to `new AnthropicDirectDeployment(new Secret(key))` inside an `AnthropicProvider`, registered with `rt.Providers.Set("anthropic", provider)` | | `ANTHROPIC_BASE_URL`, `BaseUrl` | `AnthropicOptions.BaseAddress`, or a catalog preset | | `MaxRetries`, `Timeout`, `WithOptions(...)` | `RetryPolicy` on the provider options (`AnthropicOptions.Retry`; honours `Retry-After`); `Runtime.AgentRetry` for the loop; a `CancellationToken` on every call. See [errors and retries](https://recursiveai.org/onellm/docs/errors-and-retries/). | | Exceptions: bad request, unauthorized, rate limit, 5xx | typed errors: `AuthException`, `RateLimitException`, `ContextOverflowException`, `ContentFilterException`, `RefusedException`, and so on; see [errors and retries](https://recursiveai.org/onellm/docs/errors-and-retries/) | | Response validation | unknown response fields are kept as extension data rather than failing | | `AsIChatClient("claude-...")` | `rt.AsChatClient("claude-sonnet")` in `RecursiveAI.OneLlm.Extensions.AI` | | Message Batches (`Messages.Batches`) | `IBatchProvider` on the Anthropic provider | | Token counting | `ITokenCountingProvider.CountTokensAsync`, or `rt.TokenCounter` | | Extended thinking, tool use, prompt caching | reasoning parts with their signatures are replayed on the next turn; `[Tool]` methods; `cache` on messages | | `AnthropicBedrockClient`, `Anthropic.Vertex` packages | `BedrockDeployment` and `VertexAnthropicDeployment` inside the same package; see [deployments](https://recursiveai.org/onellm/docs/deployments/) | | OpenTelemetry spans with `AddSource("com.anthropic.sdk.csharp*")` | `AddSource("OneLlm")`, `AddMeter("OneLlm")`, for every provider alike | ## Google GenAI (Gemini) The Gemini documentation pages this table was written from show their client samples in Python, JavaScript, Go and Java, not .NET; for the .NET library, read its own documentation for method names. The Gemini API operations those pages document map as follows. | Gemini API documents | OneLLM | |---|---| | `generateContent` | `Llm.AskAsync("gemini-flash", ...)`; the `GeminiProvider` speaks `generateContent` | | Streaming | `Llm.StreamTextAsync` / `Llm.StreamAsync` | | Function calling | `[Tool]` methods in an agent; thought signatures are kept and replayed | | Structured outputs | `Llm.AskAsync` | | Thinking | `thinking:` in a definition, `AskRequest.Reasoning` | | Embeddings (`gemini-embedding-001`, `gemini-embedding-2`) | `Llm.EmbedAsync("gemini-embedding-001", ...)` | | Token counting | `ITokenCountingProvider.CountTokensAsync` on the Gemini provider | | Context caching | `cache` on messages; see [Gemini](https://recursiveai.org/onellm/docs/gemini/) | | Batch API | `IBatchProvider` on the Gemini provider | | Image generation, text to speech | `IImageProvider`, `ITextToSpeechProvider`; the Interactions API and Live API are not implemented | | Vertex AI | `VertexGeminiEndpoint` with `GoogleBearerAuth`; see [deployments](https://recursiveai.org/onellm/docs/deployments/) | | OpenAI compatibility endpoint | not needed: use the Gemini provider; or an OpenAI-compatible preset can point at any compatible URL | ## Sources Read for this page (public documentation, read for interoperability under each site's terms; no code copied): - `https://learn.microsoft.com/en-us/dotnet/ai/microsoft-extensions-ai` and `https://learn.microsoft.com/en-us/dotnet/ai/ichatclient` — Microsoft Learn, site terms. - `https://learn.microsoft.com/en-us/agent-framework/overview/` — Microsoft Learn, site terms. - `https://developers.openai.com/api/docs/libraries` — OpenAI developer docs, site terms. - `https://platform.claude.com/docs/en/cli-sdks-libraries/sdks/csharp` — Anthropic docs, site terms. - `https://ai.google.dev/gemini-api/docs/libraries`, `.../text-generation`, `.../embeddings`, `.../streaming` — Google developer docs, content CC BY 4.0, code samples Apache-2.0. # Agents: the loop, approvals, sub-agents, monitors and hooks An agent is a definition run as an instance with tools. `Llm.RunAgentAsync(path, inputs)` runs one to rest and returns an `AgentResult`; `Llm.StartAgentAsync(...)` returns an `AgentRun` to follow, steer, approve and cancel; `Runtime` has the same calls plus `GetAgentAsync(instanceId)` to pick one up again, including after a restart. ## The loop 1. The system prompt and initial prompts are rendered like a scripted run's. Initial tool calls (`initialPrompts` entries with `toolCalls`) run first, through the same path as any call, marked `injected`. They stop for approval like any other call (a definition can be written by a model, so its author is not an approver), and an injected `ask_user` leaves the instance `waiting` for the answer; in both cases the task prompt is added behind the held turn and the remaining initial prompts do not run. 2. Each turn: the model is called with the definition's tools; the tool calls it makes run; the turn is saved with its results. 3. Calls run together when there is more than one and every one is `parallelSafe` (the default); a single `parallelSafe: false` call makes the whole turn run one after another in call order. Results are returned to the model **in call order**, not in finishing order. 4. The run ends in one of these ways: | Ends because | State | `AgentResult.StopReason` | |---|---|---| | a loop-ending tool ran (`task_complete`, or any tool with `endsLoop`); the rest of that turn's calls still ran | `stopped` | `Completed` | | the model answered without a tool call | `waiting` | `Waiting` | | `ask_user` asked a question | `waiting` | `Waiting` | | a call needs approval | `waiting_for_approval` | `WaitingForApproval` | | `maxIterations` (default 200) model turns were used | `error`, message `max iterations (N) reached`, no exception | `MaxIterations` | | a model call failed after its retries | `error`, the typed exception is thrown | | | the token was cancelled | `stopped`, `cancelled: true`, the partial turn kept; `RunCancelledException` | | Error rules: a tool that throws or times out gives the model an error result and the loop goes on; an unknown tool name or malformed arguments do too. A provider failure the provider calls transient is retried `Runtime.AgentRetry.MaxRetries` times more (default none: provider clients already retry), honouring `Retry-After`. Every step is saved to the instance store and written to the record sinks; usage and cost are per turn, per tool call and per instance subtree (`AgentResult.SubtreeCost`). ## Approvals A tool with `RequiresApproval`, or a hook that returns `RequireApproval`, holds its call. The turn is saved with that call unanswered and an entry in `pendingApprovals`; the other calls of the turn run now. `AgentRun.ApproveAsync(toolCallId?)` and `DenyAsync(toolCallId?, reason?)` record a decision in the instance file; when all are decided the run continues. The decision survives a restart: load the instance with a new `Runtime` over the same store, call `runtime.ApproveAsync(instanceId)` and await the run. A denied call reaches the model as an error result with the reason. Shell commands use the same gate when the grant is `Capabilities.ShellWithLoopApproval`: every `execute_terminal_command` call is held with a pending approval showing the command, and runs only after `ApproveAsync`, also in another process after a restart. `Capabilities.ShellWithApproval(approver)` keeps asking its delegate inside the tool instead (the host's own prompt, as the CLI's terminal approver does). A saved instance whose grants said "approval" and that resumes in a runtime with no approver of its own uses the loop gate. ## Messages and steering `AgentRun.MessageAsync` (or `message_agent` from another agent) delivers a message. To a running agent it is read at the next turn boundary, after the results of the turn in progress. To a `waiting` or `stopped` agent it starts a new run segment. While approvals are pending it waits until they are decided. A message from the person or host (`MessageAsync`, source `api` or `user`) sent to an agent that is waiting on `ask_user` is the answer: it becomes the result of that call. Messages from other agents or the runtime (a monitor alert) join the history behind the question and do not answer it. A message to an instance in `error` starts a new run segment, as it does for any resting instance. ## Sub-agents `run_agent_async` starts another instance in the same `Runtime`, linked by `spawnerInstanceId`, `callerInstanceId`, `callerToolCallId` and `delegationPrompt`. The child's grants are the parent's grants intersected with what the caller asks for, so a child never holds more than its parent (`Capabilities.Intersect`). A definition file is read only from under the caller's file roots, after symlinks are followed; definitions registered on `Runtime.AgentDefinitions` need no file access. The cost of children rolls up into the parent's subtree. - **Tools.** A child started from a definition *file* holds at most the tools its parent holds (plus `task_complete`): a reference to any other tool fails the start with a message naming the tool. A model can write such a file, so the file cannot widen the set. A definition the host registered on `Runtime.AgentDefinitions` chooses its own tools, whatever its parent holds; its grants are still the parent's at most. Registering a definition therefore lets any agent that can name it use the tools it lists, including a shell tool the caller lacks (the child still runs under the caller's grants). - **Reach.** The agent tools (`query_agent`, `message_agent`, `run_agent_async` with `agentId`, `await_agent`, `monitor_agent`, `list_agent_instances`) work on the calling agent and the agents it started, found by following `spawnerInstanceId` (live or saved), because another instance may hold wider grants than the caller. A monitor belongs to the agent that armed it. Set `Runtime.AgentsMayReachAnyInstance` to let agents reach every instance in the store; calls on the `Runtime` itself are never restricted. - **Grants survive a restart.** The grants an instance runs with are saved in its file (the `capabilities` key, see the instance format). When any runtime resumes it (a message, an approval, a lazy load), it gets those grants intersected with that runtime's, so a read-only start stays read-only and a runtime never widens a saved instance. The key is written with a marker (`capabilitiesRecorded`). An instance file with neither (written before grants were recorded) gets the resuming runtime's grants. A file with the marker but no valid `capabilities` entry (missing, or not a well-formed summary) fails closed: it gets no grants and refuses to run, with an error naming the instance. The marker is not a signature; the instance store must be out of the model's reach, which the runtime arranges for its file store directory. Likewise `spawnerInstanceId` in a saved file is trusted as written. ## Agent tools | Tool | What it does | |---|---| | `task_complete` | ends the loop with `result` and `summary` (and an optional `title`) | | `run_agent_async` | starts an agent (`agentSwarmUri`, `prompt`) or messages one (`agentId`) and returns its instance id | | `await_agent` | waits for one or all of `ids`; reports result, cost and context use of each; any message for the caller ends the wait | | `message_agent` | delivers `prompt` to the instance `agentId`; with no arguments it points at `run_agent_async` | | `monitor_agent` | arms a monitor on an agent (below), or just reports its status | | `await_monitor` | waits for monitors to fire | | `list_agent_instances` | lists saved instances, filtered by name, state or swarm | | `query_agent` | searches an instance's messages | Their names and input properties are the ones existing agent definitions use, so those definitions run unchanged. They sit in `Runtime.Tools` beside the built-ins (`docs/capabilities.md`); a reference such as `/agent-tools/os/read-file` or `/runtime/agents/await-agent` finds its tool by the last path segment. A host or a package adds more with `Runtime.Tools.Add` (for example a decider tool). ## Monitors `monitor_agent` takes up to four conditions, combined with OR: `contextAt` (a token count such as `400k`, or a percentage of the window), `turns` (more turns from now), `onFinish` (the agent rests) and `elapsed` (`30m`, `2h`). The first to hold fires the monitor, once, and disarms it; arming again replaces it. A fired monitor sends the watcher a runtime-sourced message, unless the watcher is blocked in `await_monitor`, which returns the alert instead. `elapsed` only reports: it never stops the agent. `Runtime.AgentEvents` raises a `MonitorFired` event. ## Context strategies See `contextStrategy` in `spec/definitions.md`: `clearToolResults`, `compact`, `providerNative`. Each run is recorded on the instance with what it replaced and what it cost. ## Hooks `Runtime.Hooks` takes `IHook` implementations or lambdas (`BeforeInference`, `AfterInference`, `BeforeToolCall`, `AfterToolCall`, `BeforeCompaction`, `OnMessage`, `OnStateChange`, `OnCost`); an `IHook` can also observe `StreamEvent`s. | Point | Decisions it can return | |---|---| | `BeforeInference` | Continue, `ModifyRequest`, `Block` (the run fails with `HookBlockedException`), `Respond` (answer without calling the model) | | `AfterInference` | Continue, `ModifyMessage`, `Block` | | `BeforeToolCall` | Continue, `ModifyInputs`, `Block` (the model sees an error result with the reason), `RequireApproval`, `Respond` (a result without running the tool) | | `AfterToolCall` | Continue, `ModifyResult`, `Block` | | `BeforeCompaction` | Continue, `Block` (skip this compaction) | | `OnMessage`, `OnStateChange`, `OnCost`, `StreamEvent` | observe only | Hooks run in registration order; host hooks wrap hooks a definition names (those come with the `.JavaScript` package). **Interceptors fail closed:** one that throws blocks the step. **Observers fail open:** one that throws is recorded and the run goes on. Every Modify, Block, Respond and RequireApproval is written as a `hook` record. With no hooks registered a run builds no hook context and allocates nothing for them. # Capabilities and the built-in tools What a tool may touch is decided by the host, never by a definition. The host passes a `Capabilities` value to the `Runtime`; every built-in tool asks it before it reads a file, starts a process or opens a connection. A definition's `references` only name tools; if a reference needs a grant the runtime lacks, the definition fails to load (strict, the default) or the reference is dropped with a warning (lenient). ## Composing grants ```csharp var caps = Capabilities.Files("work") // read and write under ./work + Capabilities.ReadOnlyFiles("/data/reference") // read only + Capabilities.ShellCommands("git status", "dotnet") // allow-listed, no shell + Capabilities.Network("api.example.com", "*.docs.example.com"); ``` | Grant | Default | Notes | |---|---|---| | Files | the working directory, read and write | read roots and write roots are separate | | Shell | off | `Capabilities.Shell` (open), `ShellCommands(...)` (allow-list), `ShellWithApproval(approver)` (every command asks that delegate inside the tool), `ShellWithLoopApproval` (every command is held in the agent loop's saved `waiting_for_approval` gate until a person approves it, also after a restart) | | Network | off | `Network(entries)`; `AnyPublicHosts` allows every public host | `a + b` unions the grants. The grants an agent started with are saved with its instance and re-applied (intersected with the resuming runtime's) on resume; network entries are saved only for the built-in `Capabilities`, so a custom `ICapabilities` that allows network resumes with it off. Agents reach only their own sub-agents through the agent tools (see the agents notes). A saved instance whose grants were recorded but are missing or malformed fails closed: it gets none and does not run (see the agents notes). The saved `spawnerInstanceId` and grants are trusted when an instance is loaded; they carry no signature. They are protected by keeping the store out of the model's reach (the protected paths above), not by checking them, so a host that puts an instance store somewhere the file tools can write has no isolation between agents. Host-registered definitions (`Runtime.AgentDefinitions`) keep their own tool sets, bounded only by the parent's grants: registering a definition lets any agent that can name it use its tools. `a.Intersect(b)` is what a sub-agent gets: never wider than either side (roots are intersected by containment, the shell is the stricter policy, hosts must be allowed by both, and with two approvers both are asked). An intersection never reaches a kind of address one side lacks: `AnyPublicHosts` intersected with `Network("localhost")` allows nothing, because public-only access never covers loopback, private or link-local addresses, and entries that can (addresses, ranges, `localhost` names) are dropped. Write protections (below) are the union of both sides'. ## Files Every path is resolved to a real path before the root check: relative paths start at the working directory, `..` is applied to the real directory (so `link/..` is the link target's parent), symlinks are followed (loops are refused), and a path whose last components do not exist yet (a write target) is resolved through its deepest existing ancestor. A path outside the roots, a sibling that merely shares a prefix (`/work2` for `/work`), and a symlink or dangling symlink leading out are refused. On Windows and macOS comparison ignores case; Windows device names (`CON`, `NUL`, …) and alternate streams are refused; junctions and other reparse points are resolved like symlinks. **What is guaranteed between check and use.** Files are opened through a double check: resolve, open, then resolve the opened path again and compare. A link swapped in between the first resolution and the open is detected and the handle is closed before anything is read, and a write through a swapped link does not truncate the outside file. Not covered: a link swapped and swapped back around the open, hard links (a hard link inside a root is a name for the same file), and a dangling-link swap on a write target can create an empty file outside the roots before the refusal. Hosts that face hostile local users should place the roots on a directory only the agent can write to. **Files the model cannot write, or read, even under a root.** Two kinds of path are refused by the file tools: - *Repository control paths*: any `.git`, `.hg`, `.svn`, `.bzr`, `.jj`, `_darcs`, `.fslckout`, `.sl` or `CVS` component (case-insensitive, trailing dots and spaces ignored) and `.gitmodules`. A root that is itself one of these, or itself a git directory, is protected too. A program on the shell allow-list (`git`, a build tool) reads configuration and hooks from there, so a write is a way to run any command. Writes only; reads are fine. A git directory is also recognised by what it holds, not by its name: any directory below a root that has `HEAD`, `objects` and `refs` (a bare repository, or the target of a working tree's `.git` file made with `--separate-git-dir`) refuses writes to everything in it, and the write that would complete that set is refused whatever order the files arrive in. (A `config` planted in a plain directory that a later `git init` adopts is not covered; do not allow-list `git init` next to a model that writes files.) **Launchers and git front ends.** An entry counts as listing git when any of its words resolves (by name, or through PATH and symlinks) to `git` or a `git-*` helper, so `env git status`, `timeout 60 git log`, `nice git ...` and `/usr/bin/env git ...` arm the rule below. For safety the rule is also armed, without looking further, by an entry whose program is a launcher (`env`, `nice`, `ionice`, `timeout`, `nohup`, `xargs`, `stdbuf`, `time`, `command`, `exec`, `setsid`, `chroot`, `sudo`, `doas`, `watch`, `parallel`) or a program that runs git itself (`gh`, `hub`, `gitk`, `tig`, `lazygit`, `make`, `npm`, `npx`, `yarn`, `pnpm`, `cargo`, `pip`, `go`, `dotnet`, `find`). A bare launcher entry runs any program, git included, so it is arbitrary execution on its own. This is a list of well-known names, not a proof: any other program that runs git or loads its configuration needs `Capabilities.WithGitCeiling()` on the runtime's grants. The cost is that `config` files and `hooks/` or `info/` paths become unwritable for the tools when such an entry is listed. **Git files by role.** (This holds for the whole runtime, not just the writer: a runtime whose shell allow-list names git marks its grants, the mark survives `+` and `Intersect`, so a child that asked for files only, or whose intersection dropped git, cannot plant the files a git-listed sibling would later run. Two separate runtimes sharing one tree are not linked; give the one without git `WithGitCeiling()`, or keep the trees apart. An entry counts as git when its program is `git` or a `git-*` helper, or when it resolves, through PATH and symlinks, to one.) When `git` is on the shell allow-list (any entry whose program is `git`), the file tools also refuse, in any directory, a file named `config`, `config.worktree`, `commondir`, `gitdir` or `.gitconfig`, and anything under a directory named `hooks` or `info`. The content rule above cannot see a directory that is still empty, and a listed program (`git init --separate-git-dir`, `mkdir`) can complete it around a config the model planted; git then runs the `core.fsmonitor` or hook command it names. The role rule is deliberately tied to git being listed: with no git on the list nothing executes these files, and an application's `config` or `hooks/` directory stays writable. It does over-refuse for a git user (an application `config` file cannot be written by the tools), and `WithRepoControlWrites()` lifts it. The `hooks` and `info` rule is deliberately broad (it also refuses `src/hooks/useAuth.ts`): narrowing it to git-shaped trees would need the directory to already look like one, which is the check an attacker defeats by creating the pieces in another order. A `HOME` inside a root makes `.gitconfig` there live, so keep `HOME` outside the roots when git is listed. **Allow-listing `git` beside programs that create directories, repositories or files (`git init`, `git clone`, `git worktree`, `mkdir`, `cp`, `mv`, `touch`, an editor) is equivalent to arbitrary code execution**, because those can place a config or hook where git will read it. Every path component counts for the control-name and git-directory tests, including those above the root, so a root below a control directory (`.git/hooks`, a bare repository's `hooks` or `refs/heads`) is protected too. `Capabilities.WithRepoControlWrites()` allows all of this when the host has no shell granted or trusts the model. `apply_patch` checks every target, counting the files the patch itself creates, before writing any, so a refused file leaves all files untouched; only a failure while writing (a full disk, a concurrent change) can leave a partial result. - *The runtime's own state*: the `.onellm` directory under a `Capabilities.Files(root)` grant, and the directories of a `FileInstanceStore` and `JsonlRecordSink` the `Runtime` uses (read `Runtime.Capabilities` to get the protected value), plus anything added with `Capabilities.WithProtectedPaths(...)`. These are neither readable nor writable by `read_file`, `write_file`, `replace_string`, `apply_patch`, `list_files` (the directory is left out of listings), `search_files_regex` and `run_decider`: the saved instances hold other agents' transcripts and tool results, which would undo the subtree scoping of the agent tools, and the model could rewrite its own transcript, approvals and audit records. `ReadOnlyFiles` protects the default `.onellm` too. The `todo` and `memory` tools still read and write their files in the default state directory; if you move it with `BuiltinOptions.StateDirectory`, protect it too. ## Shell `execute_terminal_command` is off until granted. - **Open:** the line runs in the platform shell. - **Allow-list:** the line is never given to a shell. It is split into a program and arguments; shell syntax (`;`, `&&`, `|`, backticks, `$(`, redirection, globbing, quoting tricks) is refused before anything runs, and metacharacters inside an argument are plain text. An entry is a program (`ls`) or a program with leading arguments (`git status`), which then allows only those subcommands. The program is looked up on `PATH` only (never the working directory). - **Environment.** Allow-listed and approval-gated commands start from a minimal environment (`PATH`, `HOME`, `USER`, `LOGNAME`, `LANG`, `LC_*`, `TERM`, `TZ`, `TMPDIR` and the Windows system variables); nothing else of the host's environment reaches them. The host adds names with `BuiltinOptions.ShellInheritedEnvironmentNames`. The model may not set any variable (`GIT_CONFIG_*`, `PAGER`, `HOME`, `PYTHON*` and the like change what an allowed program runs) unless the host names it in `BuiltinOptions.ShellModelEnvironmentNames`. With an open shell the child inherits the host environment minus variables that look like secrets (names ending in KEY, TOKEN, SECRET or PASSWORD, cloud credential variables; `BuiltinOptions.ShellInheritSecrets` keeps them), which is a name heuristic, not a guarantee; the model may set any variable except those that replace the program search path or loader (`PATH`, `LD_*`, ...). - **An allow-list entry is not a sandbox.** It grants everything that program can do. `git`, `dotnet`, `make`, `npm`, `find` and interpreters can run arbitrary code from files in the tree or from their arguments; list only programs and subcommands whose full behaviour you accept. The file grants do not apply to what a listed program opens: `cat /abs/path` reads any file the operating-system user can, and `working_dir` needs only read access. - **Terminal approval** (the `onellm` tool) prints the program, arguments and working directory with control characters shown as `\uXXXX` escapes. - **Output:** 10 KB per stream by default (`BuiltinOptions.ShellOutputLimitBytes`): the head and the tail are kept with an elision marker. Default timeout 30 s, capped at `ShellMaxTimeoutSeconds`. Timeout and cancellation kill the process and its children. - Known gap: the kill is `Process.Kill(entireProcessTree)`; a child that detaches into its own session (`setsid`) can outlive a timeout. It needs an open shell, because a program on an allow-list cannot be given shell syntax. - Known gap: hard links. A hard link inside a root to a file outside it reads and writes the same file; .NET has no portable link count without native calls. Keep roots on a directory only the agent writes to. ## Network Tool traffic goes through `EgressClient`, which is public so adapters can reuse it. Entries: `host`, `*.suffix` (subdomains, not the apex), `host:port`, an address or a CIDR range, or `localhost`. Rules: - A name entry never grants loopback, private or link-local addresses, even if the name resolves to one (DNS rebinding); the address is checked when the connection is made. An address or CIDR entry, or the `localhost` entry, does grant them. - Redirects are followed by hand, at most a fixed number of hops, and every hop is checked against the grant before it is requested. When a hop changes origin, every caller header except Accept, Accept-Language, Accept-Encoding, Content-Type and User-Agent is dropped, and a 307 or 308 that would resend a request body there is refused. A 303 turns a POST into a GET. - Connection pools are kept per network policy. The address check runs when a connection is opened, so a pool shared between grant sets would let a narrower grant reuse a loopback connection a wider one opened; grants with the same entries share a pool, different ones never do. - Loopback, private and link-local ranges include the IPv6 forms that embed or translate to IPv4 (`::ffff:a.b.c.d`, `::a.b.c.d`, `::ffff:0:a.b.c.d`, `64:ff9b::/96`, `64:ff9b:1::/48`, `2002::/16`). - The media tools (`generate_image`, `transcribe`, ...) read granted files and send them to the provider the host configured; the network grant is not consulted for that provider. - System proxies are not used. ## Built-in tools | Tool | Grant used | |---|---| | `read_file`, `list_files`, `search_files_regex` | file read (`read_file` of a URL: network) | | `write_file`, `replace_string`, `apply_patch` | file read and write | | `execute_terminal_command` | shell | | `http_request`, `search_web`, `fetch_webpage_markdown` (`.Web` package) | network | | `todo`, `memory` | write root (state under `.onellm/` of the working directory by default) | | `ask_user` | none | `search_files_regex` uses a linear-time regex engine, so a pattern cannot hang the tool. Glob patterns (`inclusionPatterns`, `exclusionPatterns`, `.gitignore` lines) are matched in time proportional to pattern times name, however many `*` they hold. `list_files` marks a symlink as `(symlink)` and does not print its target, which may lie outside the roots. `run_decider` resolves every link in the decider's directory against the read roots before loading it. `search_web` is costed per query from the pricing data in `SearchPricing`; the Brave key comes from `BRAVE_API_KEY`, is held in a `Secret` and is scrubbed from errors. `ask_user` returns one JSON part `{"type":"ask_user","question":…,"choices":[…]}`; `AskUser.TryGetRequest` recognises it so a runner can put the instance in a waiting state. ### `apply_patch` format A unified diff, as `diff -u` or `git diff` print it: - File headers `--- a/path`, `+++ b/path` (a leading `a/` and `b/` are removed when both are present); `/dev/null` as the old path creates a file and as the new path deletes one. Renames are not supported. - Hunks `@@ -oldStart,oldCount +newStart,newCount @@` with lines starting with a space, `-` or `+`; `\ No newline at end of file` is honoured. - Line numbers are hints: a hunk applies where its context and removed lines match exactly, nearest the stated line. If any hunk matches nowhere the whole patch is rejected and nothing is written. The file's line endings are kept. `dry_run` checks without writing. ## Reference loading `ToolReferenceResolver` checks a definition's references against the registered tools and the grants. `Strict` throws `BadDefinitionException` (file, line, column, naming the tool and the grant or package to add); `Lenient` removes the reference and records a warning. A reference to a group counts as granted only when every tool in the group is granted; in lenient mode the warning says the source was left out, and the tools of the group that are granted are still registered individually (they hold only grants the runtime gave). A definition cannot grant anything. ## On Windows The same rules hold on Windows, with these forms handled explicitly (each has a test that runs there): - **Junctions and symbolic links** are followed like any link before the root check, for reads, writes, listings, searches and `apply_patch`. - **Case and folding.** Roots, protected directories and the version-control names (`.GIT`, `.Git`, `CVS`) compare without regard to case; on Windows and macOS the letters U+0131 (dotless i), U+017F (long s) and U+212A (Kelvin sign) are also folded to `i`, `s` and `k`, because some file systems treat them as those letters. CI records what each file system does; the folding stays because a false match only refuses more. - **Extended-length roots.** `Capabilities.Files(@"\\?\C:\work")` is the same grant as `C:\work`. - **8.3 short names.** Windows can answer to a generated short name (`GIT~1` for `.git`, `ONELLM~1` for `.onellm`), which would walk around name-based protection. A path component below a root that contains `~` must be the exact name of an entry in its directory; one that exists only through its alias is refused. A short alias of the root itself is a different spelling than the grant and is refused as outside; grant the root by its long name. Volumes with short names switched off have no aliases to refuse. - **Trailing dots and spaces** are dropped from each name, as Windows does, before any comparison (`.onellm.\instances` is `.onellm\instances`); a name of only dots and spaces is refused. - **Drive-relative, device, extended and network forms** (`C:foo`, `\foo`, `/foo`, `\\?\`, `\\.\`, `\??\`, `\\server\share`) are refused. A path on a drive or share no root is on is refused before anything is probed, so no network connection is made. Forward and back slashes both separate names. - **Alternate data streams** (`file.txt:stream`) and **device names** (`NUL`, `CON`, `COM0`..`COM9`, `LPT0`..`LPT9`, the superscript forms, with or without an extension) are refused. - **cmd.exe, batch files and PowerShell.** With an allow-list, a program that re-reads its arguments as commands (`cmd`, a `.bat` or `.cmd` file, `powershell`, `pwsh`) is refused any argument holding the characters that start another command or expand a variable (`& | < > ^ % !` and quotes for cmd; `; & | $` backtick, parentheses, braces and quotes for PowerShell). An entry that lists `cmd /c echo` therefore cannot be extended to `cmd /c echo hi & whoami`. A bare `cmd` or `powershell` entry (no pinned arguments) is arbitrary execution: `/c calc`, `-Command Remove-Item ...` and `-EncodedCommand ` contain none of the refused characters. The argument check only helps when the entry pins the program's own arguments (`cmd /c mytool`, `powershell -NoProfile -File x.ps1`). Listing `cmd`, `powershell` or an interpreter without leading arguments still grants everything that program can do: an allow-list entry is not a sandbox. `.bat` and `.cmd` are not found by bare name in allow-list mode (PATHEXT entries for them are skipped); name the file by its full path. - **Environment.** Allow-listed and approval-gated commands start with `PATH`, `PATHEXT`, `ComSpec`, `SystemRoot`, `SystemDrive`, `windir`, the user-profile and program-files variables, `TEMP`/`TMP` and `NUMBER_OF_PROCESSORS`; nothing else of the host's environment is passed. - **Timeouts** kill the process and its children (`Process.Kill` of the tree). Not covered on any platform: hard links (see above), a detached process that survives its parent, and an allow-listed program that is itself an interpreter. # Deciders A decider answers typed questions about some state: *which team should handle this ticket* (choice), *how urgent is it* (score), *is the customer angry* (a yes/no probability). The definition is a `decider.yaml` (format: `spec/definitions.md`); optional logic turns the caller's inputs into what the templates read and the answers into an outcome. ```csharp DecisionResult r = await Llm.DecideAsync("deciders/triage-ticket", new Dictionary { ["ticket"] = ticket }); r.Answers; // by question id r.Outputs; // the logic's outcome, or the answers when there is no logic r.Cost.Total; r.Usage; r.Instance; Routing outcome = await Llm.DecideAsync("deciders/triage-ticket", RoutingJson.Default.Routing, inputs); // outputs read into a type (AOT safe) ``` The `Runtime` has the same calls plus the run APIs: `ReplayDecisionAsync`, `GetDecisionAsync`, `ListDecisionsAsync`, `DecisionStatsAsync`. A tool that wants to run a decider calls `Runtime.DecideAsync(decider, JsonElement inputs, origin)`. ## Three engines, chosen by the model `model:` resolves through the catalog. A model of kind `decision` is a decisions engine; any other model is asked as a chat model. | Engine | Model examples | How questions are sent | |---|---|---| | TypeSafe Jev | `jev-latest`, `jev-1.14.0` | One request holds the state and every question; split by token budget (90% of the context window) when it would not fit, up to 8 requests in flight. Bytes match the reference service for the same decider and inputs. | | OpenAI Decisions | `openai-decisions` | `POST /v1/decisions`: the state is `input_json` (images become `input_image`), questions are `predicate`, `choice` or `score`. At most 400 questions per request. `cache.key` becomes `prompt_cache_key`; `cache.explicit` adds an explicit breakpoint after the state. | | Chat model | any chat model | One call per question (and per sample) over a shared system prefix that holds the instructions and the state; the model returns probabilities, and choice, score and confidence are derived from them. Off-schema replies are retried up to three times, never repaired. | Notes that follow from the engines: - A refusal from OpenAI Decisions is a typed `RefusedException` naming the questions. - An image in the inputs needs a decisions engine that takes images; on Jev it is a `CapabilityMissingException`. - Usage on OpenAI Decisions: `input_tokens` already includes cached and cache-written tokens, so they are priced once, at their own rates; output is free. - With `llm.samples` above 1 the chat engine averages the samples' distributions (temperature defaults to 1 then). - A prefix shared by the chat calls is cached where the provider supports it; the first call runs alone when the prefix is long enough to be cached. ## Inputs and templates The caller's inputs go through `prepare` (if the decider has logic), then render the `state` and the questions. A missing name is an error, not ``. A string that is exactly one action (`"{{ .inputs.customer }}"`) keeps its structure in the request; see the spec for the exact encoding. `forEach` expands a question once per element; the answer is a list; an empty list asks nothing and makes no request. If a `decider.schema.json` sits beside the definition and declares `inputs`, the inputs are checked against it first (`DeciderInputException`, with a path). The checker enforces `type`, `properties`, `required`, `additionalProperties`, `items`, `enum`, `const`, `minimum`, `maximum` and `anyOf`, and ignores every other keyword; it is a guard against wrong inputs, not a general JSON Schema validator. The same checker validates the outputs when the file declares `outputs`: what the logic returns (the answers, for a decider with no logic) is checked before the run completes, and a mismatch throws `DeciderOutputException` naming the path (`outputs.route`). `Decider.OutputSchema` and `WithOutputSchema` are the counterparts of `InputSchema` and `WithInputSchema`. ## Logic in C# Mark a partial class and add the definition as an additional file: ```xml ``` ```csharp [Decider("triage.decider.yaml")] public static partial class Triage { // optional: JsonElement Prepare(JsonElement inputs) static JsonElement Decide(Answers answers, JsonElement inputs) => JsonSerializer.SerializeToElement(new Routing(answers.Topic.Choice.ToString(), answers.Angry.Probability > 0.4), RoutingJson.Default.Routing); } DecisionResult r = await Llm.DecideAsync(Triage.Decider, inputs); ``` The generator adds `Answers` (a `ChoiceAnswer`, `ScoreAnswer` or `NoulAnswer` per question; a list for `forEach`), one enum per choice question, the `Yaml` text and the `Decider` property. It reads the questions with a small block-YAML reader and reports `ODG001` to `ODG004` when the class is not partial, the file is not an additional file, a choice's criteria are not a block map, or two names collapse to one C# name. Everything it produces is trim and AOT safe; the full definition is still validated when the decider loads. `IDeciderLogic` is the seam other packages implement (the JavaScript package runs `decider.js` through it). ## Runs Each run is an instance (`Instance.Name` is the decider's name), saved to the runtime's store and recorded to its sinks with the cost lines of every call. `origin` marks where it came from: `api` (default), `agent`, `playground`, `replay`, `eval`. Statistics skip the origins you pass in `excludeOrigins`. `ReplayDecisionAsync` re-asks a recorded run's inputs, optionally on another model, as a new instance. ## Testing `OneLlm.Testing.ScriptedDecisionProvider` answers from a function and records the requests; for chat engines use `ScriptedProvider`. The fixtures for the Jev and OpenAI request and response shapes are written from the vendors' references, not recorded from live calls. ## Cost Jev prices input tokens only; OpenAI Decisions prices uncached input, cache reads and cache writes and not output. Chat engines price as their model does. All money is `decimal`; see `docs/pricing-and-counting.md`. # Templating `OneLlm.Templating` renders prompts written in the language of Go's `text/template` package. It is a standalone engine: it knows nothing about agents or providers, takes a tree of plain values as its scope, and is used by definitions, scripted runs and deciders. Parsed templates are cached and immutable, so rendering is thread-safe. ```csharp string s = Template.Render("Hello {{ .inputs.name | upper }}", new Dictionary { ["inputs"] = new Dictionary { ["name"] = "ada" } }); // "Hello ADA" ``` ## The language The syntax and semantics are those of the Go package documentation (). Where that page leaves output open, OneLLM matches what Go prints; the behaviour was settled by running Go 1.26 against the same templates, and the cases are kept in `testdata/templating/` and replayed by the tests. - **Text and actions:** `{{ ... }}`, comments `{{/* ... */}}`, trim markers `{{- ... -}}` (a marker needs a space beside it). - **Pipelines and variables:** `a | f b`, `$x := ...`, `$x = ...`, `$` for the scope root, parenthesised pipelines, field chains (`.a.b`), index by key. - **Control:** `if` / `else if` / `else`, `range` (with `else`, `break`, `continue`, one- or two-variable form), `with` / `else with`, `define`, `template`, `block`. - **Range** works over lists and arrays, maps (keys in sorted order), and integers. A string is not rangeable. - **Builtins:** `and` and `or` (short-circuit), `not`, `len`, `index`, `slice`, `print`, `printf`, `println`, `eq` (several arguments), `ne`, `lt`, `le`, `gt`, `ge`, `call`, `html`, `js`, `urlquery`. - **Values:** `nil` or a missing key prints ``. Integers are Go `int`, other numbers are `float64` (printed as Go prints `%v`: shortest form, with an exponent below `1e-4` and from `1e+06` upward (observed from Go 1.26); JSON text switches at `1e-6` and `1e21`). Booleans print `true`/`false`. Lists and maps print as `[a b]` and `map[k:v]` with keys sorted. - **Scope values:** `null`, strings, booleans, numbers, `Dictionary` and other string-keyed maps, `List`, arrays and `JsonElement`. They are read directly, without reflection. ### Known differences from Go - `date` takes RFC 3339 times only (`Z` or `+hh:mm`, one- or two-digit hour). - Strings are UTF-16. `slice` on a string counts UTF-8 bytes, but a cut through the middle of a multi-byte character yields U+FFFD instead of a raw byte, so the length can differ. - `slice` of a list cannot extend past the list's length; Go allows it up to the backing array's capacity, which a host list does not have. - Complex number constants (`1i`) are not supported. - `%v` of a 32-bit float is not modelled (all non-integer numbers are 64-bit). - Decimal values are converted to `float64`. - Regular expressions use a linear-time engine, so back-references and look-around are not available. ## Standard functions Always available, on top of the builtins: | Function | Meaning | |---|---| | `raw s` | Returns `s` unchanged. Use `{{ raw "{{" }}` to print a literal delimiter outside a code region. | | `embed ref` | Inserts another prompt, rendered in the same scope (see [Embed](#embed)). | | `base p`, `dir p`, `ext p`, `trimExt p`, `pathJoin a b ...`, `isAbs p` | Slash-path helpers (`/` separated; `pathJoin` cleans the result). | | `lower s`, `upper s`, `trim s` | Case mapping and whitespace trimming. | | `trimPrefix s p`, `trimSuffix s p`, `replace s old new` | `replace` replaces every occurrence. | | `contains s sub`, `hasPrefix s p`, `hasSuffix s p` | Substring tests. | | `default fallback v` | `fallback` when `v` is nil or the empty string, otherwise `v`. | ## OneLLM-only extensions These do not exist in Go's package or in any prompt host that only offers the functions above. A template that uses them is not portable. | Function | Meaning | |---|---| | `toJson v`, `toPrettyJson v` | JSON text of a value. Keys sorted, no HTML escaping; pretty form indents by two spaces. | | `toYaml v` | YAML text of a value (via OneYaml). | | `join sep list` | Joins the items of a list with `sep`. | | `split sep s` | Splits `s` on `sep` into a list. | | `indent n s`, `nindent n s` | Prefix each line with `n` spaces; `nindent` also starts with a newline. | | `truncate n s` | At most `n` characters of `s`. | | `now` | The current local time from the render's `TimeProvider`. Rendering fails if none was given. | | `date layout t` | Formats an RFC 3339 string or time with a Go layout (`2006-01-02`). | | `regexMatch pattern s`, `regexReplace pattern repl s` | Regular expressions (`$1`, `${name}` in the replacement). | | `dict k v ...`, `list a b ...` | Build a map or a list inline. | Hosts can register their own functions with `TemplateFunctions` (delegates with declared argument kinds; no reflection). A host function can replace a standard one or a builtin by name. ## Scope The host supplies the scope. The names OneLLM definitions use: `.inputs`, `.system`, `.time` (`now`, `today`, `timezone`), `.git` (opt-in), `.env` (empty unless the host allow-lists names), `.swarm`, `.invocationMode`, and `.turns[i].text` / `.turns[i].json` in scripted runs. The engine itself only needs a map. ## Code regions Prompts often show template syntax or code. Before parsing, `{{` and `}}` inside Markdown code are escaped so they print literally: fenced blocks (three or more backticks or tildes, indented at most three spaces, closed by a fence at least as long) and inline spans (a run of backticks closed by a run of the same length). Text outside code regions is parsed normally. Pass `EscapeCodeRegions = false` in `TemplateParseOptions` to turn this off. Error positions refer to the original text. ## Embed `{{ embed "ref" }}` inserts the rendered text of another prompt, in the same scope, at that point. - **Sources, in order:** the first `IEmbedSource` that returns content wins. Put the relative-path source first (`FileEmbedSource`: relative to the including file, confined to a root, symbolic links resolved, absolute references ignored), then registered prompts (`PromptEmbedSource`), then any remote source such as an HTTP resolver. **No source is configured by default**, so a template cannot read files or the network unless the host allows it. - **Depth and cycles:** nesting is limited (`MaxEmbedDepth`, default 16) and a prompt that includes itself, directly or through others, fails with the chain of keys. Both errors carry the position of the `embed` call. - Embedded text is parsed and cached like any template. ## missingKey `MissingKeyMode.Default` renders a missing map key as ``. `MissingKeyMode.Error` fails and names the key. Definitions default to `Default`; deciders always use `Error`. ## Errors `TemplateException` carries `Kind` (`Parse`, `Execute`, `Embed`, `Limit`), the template name, 1-based `Line` and `Column`, byte `Offset` and the detail. Rendering is also limited to 64 Mi characters of output and honours a cancellation token. ## Prompt caching Provider prompt caches match on the exact leading text. A system prompt that renders `now`, `date` of `now`, `.time.now` or anything else that changes between calls differs every time and defeats the cache. Put changing values late in the prompt (a user turn, or the end of the system prompt), or use `.time.today` when day resolution is enough. # Tool signatures and JSON Schema This note is the contract between a C# method signature and the JSON Schema 2020-12 the model sees. Two implementations follow it and a test keeps them identical: the `[Tool]` source generator (compile time, trim/AOT-safe) and `Tool.Create(delegate)` (reflection). Other libraries that describe tools in C# can conform to the same table. Schema dialect: [JSON Schema 2020-12](https://json-schema.org/draft/2020-12) (core and validation vocabularies). Only the keywords listed below are ever written. ## A tool | Part | Rule | |---|---| | Name | `[Tool(Name = ...)]`, else the method name in snake_case with a trailing `Async` removed (`GetDocketAsync` → `get_docket`, `HTTPRequest` → `http_request`). 1–64 letters, digits, `_`, `-`. | | Description | `[Tool(Description = ...)]`, else `[Description]` on the method, else the XML `` (generator only), else empty. | | Input schema | One closed object, built from the parameters (below). | | Output schema | Written when the return type is not text, a `Part`, a sequence of parts, a `ToolResult`, a `JsonNode` or void: the schema of the return type. Not sent to the model; it types the result of a tool that ends the loop. | | Metadata | `EndsLoop`, `RequiresApproval`, `ParallelSafe` (default true), `TimeoutSeconds` (0 = none). | ### Parameters - `ToolContext` and `CancellationToken` parameters are filled in by the runtime and never appear in the schema (at most one of each). - Every other parameter is a property named exactly as the C# parameter (camelCase by convention; names are not rewritten). - **Required** unless the parameter has a default value. A nullable type (`int?`, `string?`) without a default is still required, and accepts JSON `null`. - A default value of a string, number, boolean, char or enum type is written as `"default"`. A `null` default writes nothing. - The object has `"additionalProperties": false`. A call that names an argument the tool does not declare fails with an error result naming it. `required` is omitted when empty. - Description: `[Description]` on the parameter, else the method's `` (generator only). ## Types Keyword order in every fragment is fixed: `type`, `format`, `description`, `enum`, `minimum`, `maximum`, `minLength`, `maxLength`, `pattern`, `contentEncoding`, `items`, `minItems`, `maxItems`, `uniqueItems`, `properties`, `required`, `additionalProperties`, `default`. (Models read schemas top to bottom.) | C# type | Fragment | |---|---| | `bool` | `{"type":"boolean"}` | | `sbyte` `byte` `short` `ushort` `int` `uint` `long` `ulong` | `{"type":"integer","minimum":MIN,"maximum":MAX}` with the type's range (`ulong`: 0 to 18446744073709551615). A JSON number that is fractional or out of range is refused. | | `float` `double` `decimal` | `{"type":"number"}` | | `string` | `{"type":"string"}` | | `char` | `{"type":"string","minLength":1,"maxLength":1}` | | `Guid` | `{"type":"string","format":"uuid"}` | | `DateTime`, `DateTimeOffset` | `{"type":"string","format":"date-time"}` (RFC 3339) | | `DateOnly` | `{"type":"string","format":"date"}` (`yyyy-MM-dd`) | | `TimeOnly` | `{"type":"string","format":"time"}` (`HH:mm:ss[.fffffff]`) | | `TimeSpan` | `{"type":"string","pattern":"^-?(\\d+\\.)?\\d{2}:\\d{2}:\\d{2}(\\.\\d{1,7})?$"}` (the invariant "c" form, `[-][d.]hh:mm:ss[.fffffff]`) | | `Uri` | `{"type":"string","format":"uri"}` (absolute) | | `byte[]` | `{"type":"string","contentEncoding":"base64"}` | | enum | `{"type":"string","enum":[member names in declaration order]}`; names are matched exactly | | `T?` (value or reference type) | the fragment of `T` with `"null"` added to `type` (`["integer","null"]`); an enum's `enum` list gains `null`. A fragment with no `type` (any) is unchanged. | | `T[]`, `List`, `IList`, `ICollection`, `IEnumerable`, `IReadOnlyList`, `IReadOnlyCollection` | `{"type":"array","items":FRAGMENT(T)}` | | `HashSet`, `ISet`, `IReadOnlySet` | as above plus `"uniqueItems":true`; duplicates are refused | | `Dictionary`, `IDictionary`, `IReadOnlyDictionary` | `{"type":"object","additionalProperties":FRAGMENT(T)}` | | `JsonElement`, `JsonNode` | `{}` (any JSON value) | | `JsonObject` | `{"type":"object"}` | | `JsonArray` | `{"type":"array"}` | | record, class, struct | closed object, see below | Not supported (generator diagnostic `OLT002`; `Tool.Create` throws `NotSupportedException`): `object`, interfaces other than the collections above, delegates, multi-dimensional arrays, dictionaries whose key is not `string`, `ref`/`out`/`in` parameters, open generic types, other collection types. ### Records and classes ```json {"type":"object","properties":{...},"required":[...],"additionalProperties":false} ``` - **Properties:** public instance properties with a public getter, base class first, then in declaration order. `[JsonIgnore]` (always) removes one; `[JsonPropertyName]` renames it. - **Name:** camelCase when the type's `JsonSerializerContext` uses `CamelCase` (or `JsonSerializerDefaults.Web`); as written when it uses no policy. `Tool.Create` always uses camelCase and the generator mirrors the context, so schemas agree when the context uses `CamelCase`. Other naming policies are refused (`OLT011`). - **Required:** a `required` member, a `[JsonRequired]` property, or a parameter without a default of the type's only public constructor (a positional record's primary constructor). Non-nullable alone does not make a property required. - **Property schema:** the fragment of its type, with the property's `[Description]` (or ``; for a positional record the `` of the record) and annotations. - A type that refers to itself, directly or through its members, is refused (`OLT006`). - **Serialization:** reading and writing these types goes through a `System.Text.Json` source-generated `JsonSerializerContext` that lists the type with `[JsonSerializable]`; the generator finds it in the compilation (`OLT005` when none lists the type). Enum properties need the string enum converter in that context (`UseStringEnumConverter = true`), matching the `enum` of names above. Return values of record, collection and dictionary types must be listed too (`List` as itself). ### Annotations | Attribute | Keyword | |---|---| | `[Description("...")]` | `description` | | `[Range(min, max)]` (numeric) | `minimum`, `maximum` (replace the type's range) | | `[StringLength(max, MinimumLength = n)]` | `maxLength`, `minLength` (when n > 0) | | `[MinLength(n)]`, `[MaxLength(n)]`, `[Length(a, b)]` | `minLength`/`maxLength` on strings, `minItems`/`maxItems` on arrays | | `[RegularExpression(p)]` | `pattern` | | `[EmailAddress]` | `format: email` | | `[Url]` | `format: uri` | ### Doc comments The generator reads `///` comments from the source tree itself, so they work whether or not the project generates a documentation file. `` and `` text is flattened: whitespace collapsed, `` becomes `X`, `` becomes `p`, ``/`` keep their text. `[Description]` always wins. `Tool.Create` cannot see doc comments and uses `[Description]` only. ## Results | Return type | Result | |---|---| | `void`, `Task`, `ValueTask` | no content | | `string` | one text part | | `Part` (and subclasses), a sequence of them | those parts | | `ToolResult` | passed through | | `JsonNode`, `JsonObject`, … | one JSON part | | a number, `bool`, `Guid`, date/time, `Uri`, enum (its name) | one JSON part with that value | | a record, collection, dictionary | one JSON part from the context's serializer | | `T?` that is null | the text `null` | `Task` and `ValueTask` are unwrapped. For a tool with `EndsLoop`, the returned value is also the typed result (`ToolResult.TypedResult`). ## Errors become results `ToolInvoker.InvokeAsync` never throws for a tool's own failure: | Cause | Error result | |---|---| | Missing, unknown, null or wrong-typed argument | `Argument 'items[2].qty' must be an integer…` (the path names the argument) | | Exception in the tool | `Tool 'x' failed: InvalidOperationException: message` (no stack trace) | | Timeout (`TimeoutSeconds`, measured on the context's `TimeProvider`) | `Tool 'x' timed out after 5 s.` | Cancellation by the caller is not an error result: `OperationCanceledException` propagates so the run records itself as cancelled. Cost lines added with `ToolContext.AddCost` are kept on every path. ## Generator diagnostics | Id | Meaning | |---|---| | OLT001 | The containing type (and every type around it) must be a `partial` class | | OLT002 | A parameter type has no schema | | OLT003 | The return type is not supported (or `async void`) | | OLT004 | The method is generic | | OLT005 | A record/class/collection needs a `JsonSerializerContext` that lists it | | OLT006 | A type refers to itself | | OLT007 | Two tools in one class have the same name | | OLT008 | The tool name is not 1–64 letters, digits, `_` or `-` | | OLT009 | More than one `ToolContext` or `CancellationToken` parameter | | OLT010 | The containing class is generic | | OLT011 | The serializer context uses a property naming policy other than `CamelCase` or none | ## What each provider rejects OneLLM writes the schema above; each provider client down-converts for its API. Keywords below are the ones the vendors' structured-output and strict tool-use pages say are unsupported (checked against their reference pages when this note was written; vendors change them, and the provider clients are the place that follows). | Keyword | Anthropic (strict) | OpenAI (strict) | Gemini | |---|---|---|---| | `minimum`, `maximum` | rejected | accepted | accepted | | `minLength`, `maxLength` | rejected | not listed as supported | accepted but not enforced | | `minItems`, `maxItems` | only `minItems` 0 or 1 | accepted | accepted | | `uniqueItems` | rejected | not listed as supported | accepted but not enforced | | `pattern` | simple regex only | accepted | not listed | | `format` | `date-time` `time` `date` `duration` `email` `hostname` `uri` `ipv4` `ipv6` `uuid` | `date-time` `time` `date` `duration` `email` `hostname` `ipv4` `ipv6` `uuid` (no `uri`) | `date-time` `date` `time` | | `contentEncoding` | not listed | not listed | not listed | | `additionalProperties` | must be `false` | must be `false`; every property must be in `required` | accepted | | `default` | accepted | not listed | not listed | | union `"type":[…,"null"]` | accepted | accepted | accepted | ## Sources - JSON Schema 2020-12, core and validation: https://json-schema.org/draft/2020-12 (Internet-Draft, (c) IETF Trust, read for the keyword definitions only). - Anthropic structured outputs, JSON Schema limitations: https://platform.claude.com/docs/en/build-with-claude/structured-outputs - OpenAI structured outputs, supported schemas: https://developers.openai.com/api/docs/guides/structured-outputs - Gemini structured output, supported schema subset: https://ai.google.dev/gemini-api/docs/structured-output # JavaScript: tools, hooks, decider logic and `run_javascript` The `RecursiveAI.OneLlm.JavaScript` package runs JavaScript on [OneEcma](https://github.com/recursive-ai-oss/oneecma-csharp) in a sandbox. It adds four things to the core runtime: tool files, hook files, decider logic and the `run_javascript` built-in. It references the core package and OneEcma only. ```csharp runtime.Tools.Add(JsTools.Load("tools/convert-units.js")); // a tool file runtime.Tools.Add(JsTools.RunJavaScript()); // the run_javascript built-in runtime.DefinitionHooks = new JsHookSource(timeProvider: runtime.TimeProvider); // hooks: in definitions Decider d = JsDeciderLogic.LoadDecider("deciders/triage-ticket"); // decider.yaml + decider.js ``` ## The sandbox Every call builds a fresh engine, so nothing a script leaves behind (globals, prototypes, promises) reaches the next call. The engine has no file, network, process, environment, timer or module access of its own. A script sees the ECMAScript built-ins, `console`, `input` (snippets only) and `ctx`: | `ctx` member | Needs | Does | |---|---|---| | `ctx.instanceId`, `ctx.toolCallId` | – | Ids of the run and the call. | | `ctx.log(...)` | – | Writes a line to the tool log. | | `ctx.readFile(path)` | `fileRead` | Reads UTF-8 text (at most `MaxFileBytes`). | | `ctx.writeFile(path, text)` | `fileWrite` | Writes UTF-8 text, creating folders. | | `ctx.fetch(url, { method, headers, body })` | `network` | Resolves to `{ status, ok, headers, text, truncated }`. | | `ctx.addCost({ counter, quantity, unitPrice, per?, multiplier?, label?, source? })` | – | Adds a cost line to the call. Pass prices as strings for exact decimals; a number is read through its shortest text. Quantity, unit price and multiplier must not be negative or non-finite, text fields are at most 256 characters, and one run adds at most `MaxCostLines` (100) lines; anything else throws a `TypeError`. | Each function checks twice: the tool must have **declared** the capability (`requires`), and the host must have **granted** it. File paths go through the same guard as the built-in file tools (symlinks resolved, re-checked after the open); `fetch` goes through the egress policy (hosts checked before sending, every redirect hop re-checked, loopback and private addresses refused unless granted). An undeclared capability fails with `'fileRead' in the tool's requires list`; an ungranted one fails with the usual "is not available" message. Nothing a definition says grants anything. ### Limits All are in `JsSandboxOptions`; a limit ends the run with a `JsSandboxException` that script cannot catch. | Limit | Default | Kind | |---|---|---| | `Fuel` (steps: calls, loop iterations, regexp backtracking, long built-ins) | 200 million | `Fuel` | | `MaxAllocatedBytes` (bytes the engine allocated over the run, not live bytes) | 256 MiB | `Memory` | | `MaxHostAllocatedBytes` (live growth of the heap since the run started; backstop) | 1 GiB | `Memory` | | `TimeLimit` (wall clock on the call's `TimeProvider`; a tool's `timeout` shortens it) | 10 s | `Time` | | `MaxCallDepth` | 1,000 (catchable `RangeError`) | `Script` | | `MaxStringLength` | 16 million UTF-16 units | `Script` (`RangeError`) | | `MaxFileBytes`, `MaxFetchBytes` (one file or body) | 1 MiB | error to the script | | `MaxHostResultBytes` (all file text and response bodies one run holds, two bytes a character) | 32 MiB | `RangeError` to the script | | `MaxWriteBytesPerRun`, `MaxWritesPerRun` | 8 MiB, 256 files | `RangeError` to the script | | `MaxFetchesPerRun` | 16 requests | `RangeError` to the script | | `MaxLogChars` (`ctx.log`; one notice, then dropped) | 64,000 | – | | `Limiter` (see below) | shared: 8 runs, 2 GiB | `Busy` | | `MaxResultChars` (JSON returned) | 1 million | `Result` | | `MaxCostLines` (`ctx.addCost` calls per run) | 100 | script `TypeError` | | `MaxOutputChars` (console output; head and tail kept) | 10,240 | – | **Memory is a budget, not a ceiling.** The engine charges growth a script controls to `MaxAllocatedBytes` before allocating: array storage, typed arrays and buffers, joined and stringified text, property storage, closures, errors, promises, environments, collection entries, the symbol registry and queued jobs (a queue entry is given back when it runs). The array, buffer, string, `Map`, `Set`, `Symbol.for` and promise-queue bombs we test (40 patterns) all end in a `Memory` limit, a `RangeError`, or a `Fuel` limit from the engine alone. The budget counts bytes requested over the run, not bytes alive, so live memory can pass it: in our runs the largest peak was about twice the 256 MiB budget (object property storage and `Map`/`Set` growth), and OneEcma documents about 1.1x on x64 and under 2x on macOS arm64 for its own accounting. Size the process, not just the option, for that. The host backstop reads how much larger the .NET heap is than when the run started and ends the run when that growth passes `MaxHostAllocatedBytes` (default 1 GiB, about four budgets). It looks every time the thread has allocated a sixteenth of the limit, and only when the collector has run since the last look, so a loop that allocates gigabytes of short-lived objects and keeps none (a busy-wait on `Date.now()`, say) is not counted: churn is not memory. A growth over the limit is confirmed after a full collection before the run is ended. The heap is the whole process's, so scripts that overlap see each other's growth, and so does any other work the host does while they run; that errs on the side of stopping. It is taken between steps, so one very large single allocation can still happen before it is seen, and OneEcma 1.1.1 charges what it used to miss: collation keys (96 bytes a character), host-returned strings at the boundary (about 4 bytes a character), and a .NET out-of-memory becomes an uncatchable `Memory` limit. In our probe six parallel 15-million-character `localeCompare` runs, which peaked at 12.7 GB, now stop in about 200 ms at the allocation budget with a 415 MB peak. Live heap can still pass a budget by about 1.1x on x64 and under 2x on arm64 (OneEcma documents it), and `localeCompare` on strings above about 4 million characters each is refused by the engine. A script that holds memory without allocating more is not bounded by either. **The time limit holds even when the engine does not yield.** The engine checks the clock between steps and, since OneEcma 1.1.0, inside BigInt multiplication and exponentiation, but only between chunks of work: since OneEcma 1.1.1 a BigInt multiply stops about a second after it starts and fuel scales with the length of long-string comparisons, sorts and collation, so in our probes (eight scripts comparing 15-million-character strings in a loop, `localeCompare` loops, BigInt squaring) the engine ends every run by itself and nothing is abandoned. The host therefore races every run against the time limit plus 250 ms, and against the caller's token. When the race is lost the caller gets the `Time` failure (or the `OperationCanceledException`) at once, and the engine's thread is abandoned: its eventual result or error is discarded. The cost is real: that thread keeps using a CPU core until OneEcma can interrupt the operation (the run is closed, so the thread can no longer act and stops at its next clock check). `JsSandbox.AbandonedRuns` counts runs abandoned in the process and `JsSandbox.AbandonedRunsStillRunning` those still executing, so a host can watch for it and stop accepting such scripts. A script that stops by itself within the 250 ms grace is reported as usual, with its position. The run is started on a thread of its own (not a thread-pool thread, so a few busy scripts cannot starve the host's other work, including hooks), so a long script never blocks the caller, and cancelling the caller's token stops it. **Error explanation is on.** The sandbox leaves `ExplainErrors` at its default, so a failed run is replayed with tracing and the report gains where a value came from, "did you mean" suggestions and frame locals. OneEcma 1.1.0 gives that replay a deterministic work budget; past it the report says `explanation unavailable: the explanation exceeded its work budget` and keeps the plain report. (OneEcma before 1.1.0 had no such budget, which is why the sandbox once turned it off.) ### A run that has ended acts no more Every host function (`ctx.readFile`, `writeFile`, `fetch`, `addCost`, `log`) first checks that its run is still open, and a run is closed on every way out: it returned, threw, hit a limit, was cancelled, or was abandoned because the engine did not stop when asked. A closed run's functions throw `the run has ended`, and a request still in flight is cancelled. Closing waits (up to a second) for a function that was already running, so by the time the caller has the result nothing is still acting for it: no file is written, no cost line is added to the `ToolContext`, nothing is logged and no request lands afterwards, including a `ctx.fetch` the script never awaited. A script the host gave up on also stops itself: once its run is closed the clock the engine reads for its time limit reads far past any limit, so the thread ends at its next check instead of running on. ### Many runs at once A model can start many runs in one turn (`run_javascript` and JS tools are parallel-safe), so a host bounds them together with a `JsRunLimiter`: ```csharp var limiter = new JsRunLimiter(maxConcurrentRuns: 4, maxTotalBytes: 1L << 30, maxWait: TimeSpan.FromSeconds(5)); var tool = JsTools.RunJavaScript(new JsSandboxOptions { Limiter = limiter }); // share one limiter across a runtime's sandboxes ``` Sandboxes that name none use `JsRunLimiter.Shared` (8 runs, 2 GiB, a 10 s wait). A run waits up to `maxWait` for a slot and then fails with `JsSandboxFailure.Busy`; it never waits for ever. A run the host abandoned keeps its slot until its thread has really ended, so stuck scripts cannot pile up threads past the limit. The byte budget counts the larger of the file text and response bodies runs hold (they live in the heap, so they are not added to the growth a second time) and the largest live growth any active run has measured; the run that passes it fails with `Busy`. OneEcma charges the same strings to each run's own `MaxAllocatedBytes`; the limiter's count is the cross-run view of them, and `MaxHostResultBytes` is the explicit per-run quota on file and response text. One caller cannot take every slot. A caller is the root of an instance tree: sub-agents count against the instance that started them (the limiter follows `CallerInstanceId`), so starting children does not multiply the cap. Hooks and decider logic may use all of them, but tools and `run_javascript` leave `reservedForInterceptors` (default 2) free, so interceptors do not fail closed while a flood of tool runs holds the rest, and one caller (an instance id) holds at most `maxRunsPerCaller` (default 3) of the others. Abandoned runs count against their caller until the thread ends, Hook calls are capped per owner too (`maxInterceptorsPerOwner`, default 2), calls of one decider's logic share a cap per decider and decider logic as a whole may use half the slots, and one slot always stays free for hooks of other owners. Loading a file takes a slot as well, and a definition may name at most 16 hook files (`JsHookSource.MaxHooksPerDefinition`). Abandoned runs count against their owner until the thread ends, which is the strike rule: a caller whose scripts keep hanging is refused until they have stopped. `ActiveRuns` and `ChargedBytes` show the state. Loading a tool, hook or decider file runs its top level once, and that runs under the same guard as a run: the time limit, fuel and memory limits, and the host giving up on it if it does not stop, so a file that loops at the top level fails to load instead of hanging the loader. `ctx.readFile` and `ctx.writeFile` open only regular files. The open runs on a helper thread under a 2 s limit; a path that does not open in time (a pipe or similar waits for a peer) is refused, and the blocked open is released by opening the other end, so no thread is left behind. A special file that does open but cannot seek is refused too. ### Reproducible runs `JsSandboxOptions { Deterministic = true, FixedTime = …, RandomSeed = … }` makes `Date.now()`, `new Date()` and `Math.random()` repeat. By default the clock is the `ToolContext`'s `TimeProvider`, and randomness is unseeded. ### Errors A failure becomes a `JsSandboxException` whose message is OneEcma's compact report: the error, `file:line:column`, the source line and the most useful hint. `Kind` says whether it was a script error, a syntax error, a limit or a bad result; `File`, `Line` and `Column` are set when known. Reports name the file as given to the loader (the file's name, never its path) and carry no .NET stack. A message is cut at `JsSandboxException.MaxMessageChars` (10,240) with a marker saying how much was left out, so a script that throws a huge string cannot push it into model context or records. As a tool result the message starts with `Error: `. ## Tool files A tool file is an ES module that exports `invoke(inputs, ctx)` and describes itself in one of two ways. ```js // reshape.js: the exported `tool` object is the description. export const tool = { name: "reshape_json", description: "Picks fields out of a list of objects.", inputSchema: { type: "object", properties: { items: { type: "array" } }, required: ["items"] }, requires: [], // fileRead, fileWrite, network }; export function invoke({ items }) { return items.map(i => ({ id: i.id })); } ``` or a `reshape.tool.yaml` beside the file with the same keys (the YAML wins when both exist). Keys: `name` (default: the file's name), `description` (required), `inputSchema` (default: no arguments), `outputSchema`, `requires`, `endsLoop`, `requiresApproval`, `costHint` (USD; a string for an exact decimal), `parallelSafe`, `timeout` (seconds). `invoke` may be `async`. A string result is text; `undefined` is an empty result; anything else is a JSON part (and the typed result when `endsLoop`). A throw becomes an error result. `JsTools.Load` reads the file as the host, runs its top level once under the sandbox limits, and fails with a `BadDefinitionException` (file, line, column) if it does not parse, has no `invoke`, or its description is invalid. `JsTools.LoadDirectory` loads every `.js` file in a folder. A tool reports `ICapabilityTool.Requires` from its `requires` list, so a definition that names it is refused in strict mode when the runtime has not granted that much. Modules cannot import other files; a tool is one file. Shell commands are not available to JavaScript. `samples/js-tools/` has a unit converter (`.tool.yaml` description) and a JSON reshaper (exported description). ## `run_javascript` `JsTools.RunJavaScript()` is the built-in the model calls with `{ code, input? }`. The code runs as the body of an async function (top-level `await` and `return`), with `input` and `ctx` in scope and every `ctx` function available **where the caller's grants allow it**. The result is the console output followed by `=> `; with neither, `(no output)`. Output is capped like the shell tool's. ## Hooks Set `Runtime.DefinitionHooks` to a `JsHookSource`. A definition's `hooks: [{ file, points? }]` entries (paths relative to the definition) are read **through the run's read grants** (the same roots, symlink resolution and protected names as `read_file`), so a definition cannot load a hook from outside the directories the host granted; a refused file is a `CapabilityMissingException` that names the path the definition gave and never shows any of the file. They load as hooks that run under the host's hooks (see `docs/agents.md` for ordering and failure rules). A hook file exports one function per point it handles: `beforeInference`, `afterInference`, `beforeToolCall`, `afterToolCall`, `beforeCompaction`, `onMessage`, `onStateChange`, `onCost`. Each gets a JSON snapshot and `ctx`. **Resuming without a hook source fails closed.** A saved instance whose definition lists hooks resumes only in a runtime that can run them. Without `Runtime.DefinitionHooks` it throws `CapabilityMissingException` naming the package, whether the instance is new or resumed. The one exception is a definition whose hooks all name observer points (`onMessage`, `onStateChange`, `onCost`) and a lenient load, which skips them with a warning. `JsHookSource.LoadFile`, `JsTools.Load` and `JsDeciderLogic.Load` take a path the host chose and read it as the host. An interceptor returns nothing (continue) or `{ action, reason?, … }`: | Point | Actions and fields | |---|---| | `beforeInference` | `modify` with `system`, `maxOutputTokens`, `temperature`, `messages`; `respond` with `message` or `text`; `block` | | `afterInference` | `modify` with `message` or `text`; `block` | | `beforeToolCall` | `modify` with `inputs`; `respond` with `result` or `text`; `requireApproval`; `block` | | `afterToolCall` | `modify` with `result` (`{ text }` or `{ json }`, optional `isError`) or `text`; `block` | | `beforeCompaction` | `block` | **Interceptors fail closed:** a throw, a limit, an unknown action or a payload of the wrong shape blocks the step and is recorded. **Observers fail open:** a failure is recorded and the run goes on. Every modify, block and respond is recorded, as for hooks written in C#. An optional `export const hook = { requires: ["fileRead"] }` declares the capabilities the hook's `ctx` may use; without it a hook has none. ## Decider logic `JsDeciderLogic.Load("decider.js")` implements `IDeciderLogic` over a module that exports `prepare(inputs)` and/or `decide({ answers, inputs })`. `LoadDecider(directory)` reads `decider.yaml` and a `decider.js` beside it. Both functions are pure (no grants, fresh engine per call), may be `async`, and return JSON. A property that is `undefined` in the result is written as `null` and `undefined` itself as `null`, which is the form compiled decider bundles have always produced. A bundle built from TypeScript works as it is, because it is JavaScript. `decider.ts` works when it uses only **erasable** TypeScript: annotations, `interface` and `type`, `as`, `as const`, `satisfies`, `import type` (a module whose only import is `import type` loses it). OneEcma erases the types in place, replacing them with spaces and keeping line breaks, so errors and reports point at the original lines and columns of the `.ts` file. Nothing is type-checked. `LoadDecider(directory)` uses `decider.js` when it exists, otherwise `decider.ts`; `JsDeciderLogic.Load` takes either by extension. Syntax that cannot be erased is refused when the decider loads, with a `BadDefinitionException` that names the construct and gives file, line and column: `enum`, a `namespace` with values, constructor parameter properties, `import x = require` and `export =`, `x` casts and parameter decorators. Each has a one-line rewrite in OneEcma's TypeScript page. A `.ts` module may only import types: the logic is one file. ## Trimming and NativeAOT The package is annotated trim- and AOT-compatible: OneEcma is, and nothing here uses reflection, `Reflection.Emit` or runtime code generation (JSON goes through the core package's source-generated metadata and OneYaml). `samples/JavaScriptAotSmoke` loads a tool file, runs it and `run_javascript`, and hits a limit; it publishes with zero warnings and runs. ## What errors reveal A host function that fails (a missing file, a refused request, a bad argument) reaches the script as a plain `Error` with a short message such as `ctx.readFile: no such file`: no host stack frames, no absolute paths, no .NET exception type names. The model and the records see the same text. A result nested deeper than 64 levels is a `JsSandboxException` of kind `Result`. An instance whose recorded cost total is negative resumes with the total counted as zero. # Cloud deployments A deployment is **where a provider's requests go and how they are authenticated**. The request and response shapes are the provider's own, so the same codecs serve every deployment and each deployment is a small class. Nothing here uses a cloud SDK; signing and token exchange run on the BCL's HMAC, SHA-256 and RSA. | Provider | Deployment | Class | Auth | |---|---|---|---| | OpenAI | Azure OpenAI / Foundry | `AzureOpenAIEndpoint` (an `OpenAIEndpoint`) | `api-key` header, or a Microsoft Entra bearer token from a delegate | | Anthropic | Amazon Bedrock | `BedrockDeployment` | AWS Signature Version 4, or a Bedrock API key (bearer) | | Anthropic | Vertex AI | `VertexAnthropicDeployment` | OAuth2 access token (service-account JWT, or a delegate) | | Gemini | Vertex AI | `VertexGeminiEndpoint` with `GoogleBearerAuth` | the same | `provider.Name` stays `anthropic`, `openai` or `gemini`; `provider.Deployment` is `bedrock`, `vertex` or `azure`. Credentials are explicit values, a delegate, or (AWS only) the standard `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` and `AWS_BEARER_TOKEN_BEDROCK` variables through `FromEnvironment`. No config or credentials files, no instance metadata service and no `gcloud` state are read. Keys, session tokens and access tokens are `Secret`s and are removed from error text. ## Azure OpenAI / Foundry ```csharp var endpoint = new AzureOpenAIEndpoint(new Uri("https://my-resource.openai.azure.com/"), new Secret(key)); var provider = new OpenAIProvider(new OpenAIOptions { Endpoint = endpoint }, TimeProvider.System); ``` - **v1 form (default):** `{resource}/openai/v1/responses` (or `chat/completions`, `embeddings`, `batches`, `files`). No `api-version`; the request's `model` is the **deployment name**. - **Deployment-path form:** set `AzureOpenAIOptions.ApiVersion` (for example `2024-10-21`) and the chat, completions and embeddings routes become `/openai/deployments/{deployment}/…?api-version=…`. `Deployments` maps model names to deployment names. - **Entra ID:** pass a `Func>` and a clock. Ask your credential library for the scope `https://cognitiveservices.azure.com/.default`. The token is refreshed five minutes before it expires, and one fetch serves concurrent callers. - **Pricing:** `AzureOpenAIEndpoint.CatalogAliases` returns a catalog layer making each deployment name an alias of the model it serves, so cost and parameter rules follow the model. - **Errors:** a 400 with `error.code` `content_filter` becomes `ContentFilterException`, with the categories and severities from `innererror.content_filter_result(s)`. A 429 honours `Retry-After`. ## Amazon Bedrock ```csharp var bedrock = BedrockDeployment.FromEnvironment("us-east-1", TimeProvider.System, new BedrockOptions { InferenceProfile = "us" }); var claude = new AnthropicProvider(bedrock, TimeProvider.System); ``` - **Wire:** `POST /model/{modelId}/invoke` and `/invoke-with-response-stream` carrying the Messages body with `anthropic_version` (default `bedrock-2023-05-31`) in the body and no `model` or `stream` members. The model id is percent-encoded in the path (`:` becomes `%3A`) and, as SigV4 requires for non-S3 services, encoded once more in the canonical request. - **Streaming:** the response is the AWS binary event stream (prelude, headers, payload, two CRC-32s). Each `chunk` message carries `{"bytes": base64(event)}`; the events are the Messages streaming events, so streaming builds the same `Message` as a whole call. A bad checksum, a damaged frame or a stream that ends inside a message is a `MalformedResponseException`; exception frames (`throttlingException`, `validationException`, …) map to the typed errors. - **Model ids:** a friendly name (`claude-sonnet-4-5-20250929`) maps to `anthropic.claude-sonnet-4-5-20250929-v1:0` through a built-in table, then the rule `anthropic.`. `BedrockOptions.InferenceProfile` (`us`, `eu`, `apac`, `global`) prefixes the id for cross-region inference profiles; `ModelIds` overrides any name with an exact id or ARN. - **Auth:** SigV4 with service `bedrock`, signing `host`, `content-type`, `accept` (whole calls), every `x-amz-*` header (`x-amz-date`, and `x-amz-security-token` when there is a session token) and the body hash. A Bedrock API key is sent as `Authorization: Bearer`. - **Refused before sending** (`BadRequestException`): message batches (use Bedrock batch inference jobs) and token counting; the counter falls back to its local estimate. - **Sampling parameters:** the Anthropic codec reads `AnthropicOptions.ApiVersion` (default `2026-10-21`, an `anthropic-version` header value on the first-party API) to decide whether `temperature`, `top_p` and `top_k` may be sent: versions from `2026-10-21` on reject them, so the codec drops them. On Bedrock and Vertex the body carries its own `anthropic_version` and the header is not used, so to keep sampling parameters set `new AnthropicOptions { ApiVersion = "2023-06-01" }` (any earlier version string). - **Prices** are the first-party Claude entries; regional pricing (`region: us` multiplier) is the caller's to apply from the catalog. ## Vertex AI ```csharp var account = GoogleServiceAccount.FromFile("service-account.json"); var claude = new AnthropicProvider(new VertexAnthropicDeployment(account, "us-east5", TimeProvider.System), TimeProvider.System); var gemini = new GeminiProvider(new GeminiOptions { Time = TimeProvider.System, Auth = new GoogleBearerAuth(account.CreateTokenCache(TimeProvider.System)), Endpoint = new VertexGeminiEndpoint(account.ProjectId!, "us-central1"), Deployment = "vertex", }); ``` - **Service account:** an RS256-signed JWT (`iss`, `scope`, `aud`, `iat`, `exp` one hour out, optional `sub`) is exchanged at the token URI with the `urn:ietf:params:oauth:grant-type:jwt-bearer` grant (RFC 7523). Tokens are cached and refreshed five minutes before expiry; 5xx and 429 on the token endpoint are retried. A token delegate can replace the exchange. - **Claude:** `…/projects/{p}/locations/{l}/publishers/anthropic/models/{model}:rawPredict` or `:streamRawPredict`, `anthropic_version: vertex-2023-10-16` in the body, no `model` member, `stream` kept. The `@date` form of model ids is handled. Location `global` uses `aiplatform.googleapis.com`; others `{location}-aiplatform.googleapis.com`. - **Gemini:** `…/publishers/google/models/{model}:generateContent` (and `:streamGenerateContent?alt=sse`) over the same codec as the Gemini API. - **Gemini `countTokens` and `embedContent`:** Vertex's bodies, from the Vertex AI REST reference. `countTokens` takes `model`, `contents`, `systemInstruction`, `tools` and `generationConfig` at the top level (no `generateContentRequest` wrapper; other members are left out). `embedContent` embeds one input per call (a list of inputs is sent as several calls and joined in order), with `taskType` and `outputDimensionality` under `embedContentConfig`. - **Refused before sending** (`BadRequestException`): Claude batches and token counting; Gemini batch and files (Vertex has them only as separate job and storage APIs, which are not mapped). - Project, region and location strings are validated as plain names, so they cannot redirect a request to another host; base addresses must be https (loopback excepted, for tests). ## Conformance Fixtures are written from the vendors' reference pages and say so in their descriptions; none was recorded from a live call. Each replay checks the request path, headers and body as well as the decoded answer. Live rows need real accounts and stay open. | Deployment | chat | stream | tools | error fixtures | |---|---|---|---|---| | Bedrock Claude | `chat/basic`, `chat/model-id-with-colon` | `chat/stream` (event stream), `chat/stream-whole` | `tools/client-tools-roundtrip` | throttling, access denied, validation (too long), mid-stream exception | | Vertex Claude | `chat/basic` | `chat/stream`, `chat/stream-whole` | `tools/client-tools-roundtrip` | quota, unauthenticated | | Vertex Gemini | `chat/basic` | `stream/text` | `functions/call-and-result` | permission denied | | Azure OpenAI | `responses-v1`, `chat-completions-deployment-path` | `responses-v1-stream` | `responses-v1-function-tools` | content filter 400, rate limit 429 (+ embeddings, deployment path) | Signing is checked against the published AWS Signature Version 4 test suite (`testdata/aws-sigv4`): every case's canonical request, string to sign and Authorization header must match, except `get-header-value-multiline` (obsolete header line folding cannot be put on an HTTP request message) and `post-x-www-form-urlencoded-parameters` (the suite's own artifacts disagree with one another); for `post-x-www-form-urlencoded` the canonical request is not compared because the suite's file lists a header its own string to sign omits. The JWT path is checked against the RFC 7515 appendix A.2 vector and by verifying signatures with the public key. # OpenAI and OpenAI-compatible providers `OpenAIProvider` talks to the OpenAI API (or an Azure OpenAI / Foundry resource through `AzureOpenAIEndpoint`, see [deployments](https://recursiveai.org/onellm/docs/deployments/)) with the **Responses** protocol by default, or **Chat Completions** with `OpenAIApi.ChatCompletions`. `OpenAICompatibleProvider` speaks Chat Completions to any server that implements it. Both implement chat, streaming, embeddings, token counting (OpenAI only) and the Batch API. Streaming and non-streaming calls build equal `Message`s. ## What the Responses codec does - **Stateless.** `store: false` is always sent (`OpenAIOptions.Store` changes it). For reasoning models `include: reasoning.encrypted_content` is added so reasoning items can be replayed on the next turn. Reasoning items are replayed verbatim from `ReasoningPart.Data` when their provider is `openai`; other providers' reasoning is dropped. - **Usage.** `InputTokens` is the API's `input_tokens` minus cached and cache-write tokens, so the counters never overlap. Counters: `cacheReadInputTokens`, `cacheCreationInputTokens`, `cacheCreation30mInputTokens` and `cacheCreation12hInputTokens` (subsets of the cache-write counter), `reasoningTokens` (subset of output), `toolCalls.` (`web_search`, `file_search`, `code_interpreter`, `image_generation`, `mcp`) and image-generation token counters. The catalog declares the cache-write subsets with `pricing.subsetOf`. - **Tools.** Function tools become `function` items; free-text "custom" tools are `HostedToolDefinition("custom")` and arrive as tool calls with `{"input": text}`. Hosted tools (`web_search`, `file_search`, `code_interpreter`, `image_generation`, `mcp`) come back as `HostedToolCallPart`; an output item the codec does not know is kept as a `ProviderPart`. - **Caching.** The cache key goes to `prompt_cache_key` (64 characters at most). A TTL goes to `prompt_cache_options.ttl` (`30m` or `12h`) on models whose catalog entry has the property `promptCacheOptions`, otherwise to `prompt_cache_retention: "24h"` for a TTL of an hour or more. Turning caching off cannot be asked of the API and is reported as a warning. - **Refused, not sent:** stop sequences on Responses (use Chat Completions), reasoning efforts a model does not list, tools or structured output on models whose entry says no. **Stripped with a warning:** temperature/top_p on models that take none, reasoning settings on non-reasoning models, token budgets for reasoning. - **Warnings.** `RequestAdjuster` runs once per request; its notices, plus the codec's own, are on `ChatResponse.Warnings`. Streaming and embeddings calls apply the same adjustments but have no channel for the notices yet. ## Pricing in the shipped catalog Service tiers are `pricingMultiplier`s (flex 0.5, priority where the published ratio is clean), batch is the `batch` multiplier 0.5, and the long-context rate (above 272K input tokens) is a tier with multipliers. Web and file search calls are per-1000 items. Code-interpreter container sessions are not priced yet. Prices carry `asOf` 2026-11-05 and the pricing page as source. ## Compatible-server presets A preset supplies the base URL, the key's environment variable and the quirks recorded as catalog `properties`. Presets accept any model name. Presets for servers on the local machine (`local: true`) are priced at zero. | Preset | Base URL | Key variable | Quirks in the catalog | |---|---|---|---| | `ollama` | `http://localhost:11434/v1` | none | local | | `vllm` | `http://localhost:8000/v1` | none | local | | `lmstudio` | `http://localhost:1234/v1` | none | local | | `openrouter` | `https://openrouter.ai/api/v1` | `OPENROUTER_API_KEY` | `usage.cost` is copied to `Usage.Cost` | | `groq` | `https://api.groq.com/openai/v1` | `GROQ_API_KEY` | | | `deepseek` | `https://api.deepseek.com` | `DEEPSEEK_API_KEY` | no JSON schema, no reasoning effort | | `xai` | `https://api.x.ai/v1` | `XAI_API_KEY` | | | `mistral` | `https://api.mistral.ai/v1` | `MISTRAL_API_KEY` | `required` tool choice is sent as `any`; no stream usage; no reasoning effort | | `together` | `https://api.together.ai/v1` | `TOGETHER_API_KEY` | | Base URLs come from each vendor's documentation. The quirks for Mistral and DeepSeek, the vLLM default port and reasoning-field names are recorded from the docs only partly and are to be checked against live servers; the recorded fixtures for the presets are written from the references, so they pin the request each preset sends and how its answers decode, not the server's behaviour. ## Registering the catalog `OpenAICatalog.Register()` adds the files to `CatalogLoader`'s shipped list so a default catalog loads them; `OpenAICatalog.Shipped()` returns the layer directly for tests and custom catalogs. # Gemini provider notes The Gemini API provider (`OneLlm.Providers.Gemini`) speaks `generateContent` and `streamGenerateContent?alt=sse` on `v1beta`. These notes record how its wire shapes map onto the message model, where the API reference is silent, and what the tests cover. ## Mapping to the message model - **Turns and roles.** A user turn becomes a `user` content; an assistant turn becomes a `model` content. The results of an assistant turn's tool calls go into one following `user` content, as `functionResponse` parts in call order. - **Call ids.** Gemini may omit `functionCall.id`. The provider then makes an id (`gemini-call--`) so the loop can pair results; made-up ids are never sent back, and results are paired by name and position. - **Thought signatures.** A `thoughtSignature` on a text, call or media part is kept as its own reasoning part (`Data.signature`, `Data.on` naming the part kind) placed just before the part it belongs to, and is attached to that part again on the next request. A thought summary (`thought: true`) is a reasoning part with `Display` set and `Data.on = "thought"`. - **Hosted tools.** Code execution (`executableCode` + `codeExecutionResult`) and server-side `toolCall` + `toolResponse` become one hosted tool call each; the raw parts are kept in `Data.replay` and sent back unchanged. Google Search, Maps and file-search grounding become hosted calls with no replay data, plus `url_citation` annotations. Grounding offsets are UTF-8 byte offsets over the whole answer and are converted to character positions per text block. - **Usage.** `promptTokenCount` includes cached tokens and, with the URL-context tool, `toolUsePromptTokenCount` is billed as input. The provider reports `InputTokens = prompt - cached + toolUsePrompt`, `CacheReadInputTokens = cached`, `OutputTokens = candidates + thoughts`. Modality counters (`inputAudioTokens`, `inputImageTokens`, ...) are subsets of the input total. - **Extension data.** Unknown response fields, safety ratings, citation metadata and prompt feedback are kept under `Message.ExtensionData["gemini"]`. - **Streaming.** Both paths run the same parser, so a streamed message equals the whole-response message. Function-call arguments arrive whole in one part; the reference documents no partial-argument stream. ## Request rules - `RequestAdjuster` runs once per request against the catalog entry; its warnings, and the Gemini-specific ones below, land on `ChatResponse.Warnings`. - Gemini-specific: no switch for `parallelToolCalls=false` and no end-user identifier on `generateContent` (both dropped and noted); effort `disabled` means budget 0, `dynamic` means -1 on budget models; levels are upper-cased on level models. - Structured output uses `responseJsonSchema` unless the model's catalog entry sets `properties.schemaForm: responseSchema`, which converts the schema to the OpenAPI subset. - `ServiceTier` is the top-level request field `serviceTier` (`flex`, `priority`; the reference lists it as an optional request field and an output-only usage field). - A bad API key is answered with HTTP 400 `INVALID_ARGUMENT` (`API_KEY_INVALID`); it is mapped to an authentication error. ## Not settled by the reference - Whether a `GenerateContentRequest` inside a batch needs its own `model` field (it is omitted; the batch names the model in the URL). - The embeddings async batch (`asyncBatchEmbedContent`) is not implemented; `batchEmbedContents` is. - Streamed partial function-call arguments (see above). ## Tests `tests/OneLlm.Tests/Gemini/` holds a conformance case list (`GeminiCases*.cs`), replayed against fixtures in `testdata/providers/gemini/direct//`. Fixtures are written from the API reference (their description says so), not recorded. To rewrite them after a deliberate request change, set `ONELLM_WRITE_FIXTURES` to that directory and run the conformance tests; read the diff. The upload fixture is written by hand, because the recorder drops the upload address header. # Media: images, speech, transcription and moderation Four capability interfaces cover the non-chat endpoints of OpenAI and Gemini. Each result carries a `Cost` from the pricing engine and, where the service reports one, a `Usage`. | Interface | Method | OpenAI | Gemini | |---|---|---|---| | `IImageProvider` | `GenerateImageAsync` | `/images/generations`, `/images/edits` (input images and a mask as data URLs), partial images by SSE through `IImageStreamingProvider` | `generateContent` with the `IMAGE` response modality; one call per image when `Count > 1`, usage summed | | `ITextToSpeechProvider` | `SynthesizeAsync` | `/audio/speech`, audio streaming through `ISpeechStreamingProvider` | `generateContent` with the `AUDIO` modality: prebuilt voice, multi-speaker voices; raw PCM is wrapped as WAV unless `pcm` is asked for | | `ISpeechToTextProvider` | `TranscribeAsync` | `/audio/transcriptions` and `/audio/translations` (multipart), SSE text deltas through `ISpeechToTextStreamingProvider` | none: audio understanding is chat | | `IModerationProvider` | `ModerateAsync` | `/moderations`, texts and images in one request | none | Not implemented: Gemini's newer Interactions API for image and speech (the models are reached through `generateContent`), Gemini media streaming, video generation and realtime voice. Media in requests and results is bytes plus a content type (`ImageInput`, `GeneratedImage`, `SpeechResponse.Audio`). Settings a model does not have (for example `quality` on a Gemini image model) are dropped and reported in the response's `Warnings`. ## How usage is priced The pricing rules are in `pricing-and-counting.md`; media models use these counters. | Counter | Used by | Notes | |---|---|---| | `inputTokens`, `outputTokens` | image, speech and transcription models that bill tokens | text rates from `inputPerMillion` / `outputPerMillion` | | `inputImageTokens`, `outputImageTokens` | OpenAI and Gemini image models | subsets of `inputTokens` / `outputTokens`: the engine prices them at their own rate and takes them out of the text totals | | `outputAudioTokens` | Gemini speech models | subset of `outputTokens` | | `characters` | `tts-1`, `tts-1-hd` | UTF-16 length of the input, priced per million | | `audioSeconds` | `whisper-1`, `gpt-transcribe`, and the transcription models when a response reports a duration but no tokens | per-minute prices are written per second so the arithmetic is exact | Moderation models are free: their cost is a total of zero. Batch mode applies the model's `batch` multiplier. `gpt-4o-mini-tts` bills tokens but the speech endpoint reports no usage, so its `Cost` is null and the response carries a `usage_missing` warning; OneLLM does not estimate. A transcription response with neither usage nor a duration is treated the same way. ## Tools `MediaTools.Create(provider, imageModel, speechModel, transcriptionModel, moderationModel, options)` returns the tools the provider supports for the models given: `generate_image`, `speak`, `transcribe` and `moderate`. The host picks the model; the model of the conversation never names one. `MediaToolOptions` sets the input size limit (25 MiB), the most images per call (4), the longest text to speak (4096 characters), a cost hint, whether a person must approve each call, and the timeout (five minutes). - **Cost.** A tool adds the provider's cost lines to its `ToolContext`; the invoker puts them on the `ToolResult`. - **Outputs.** Images and audio come back as asset parts with inline data (storage `inline`); the runtime's asset store decides where they live. The tools write no files. `moderate` returns JSON with the flagged categories and scores per input. - **Files.** Paths in `images`, `mask` and `path` are read through the runtime's file grant, the same way the built-in file tools read: the path is resolved with symbolic links followed and must lie under a read root (`ICapabilities.ResolveRead`), and the file is opened with the built-ins' double check (`docs/capabilities.md`). A path outside the roots, a run with no file grant, a file over the size limit and a missing file are error results that name the argument, before any provider call. `transcribe` declares the file-read grant (`ICapabilityTool`); the other tools need no grant unless the model names an input file. The tools write no files and fetch no URLs, so the write roots and the egress policy are not involved. - **Network.** The call to the provider is the use of the network the host chose when it gave the tool a client. It is not tool network egress: `ICapabilities.IsNetworkAllowed` governs tools that open connections of their own, such as `http_request`, and is not consulted here. # Microsoft.Extensions.AI bridge `RecursiveAI.OneLlm.Extensions.AI` connects OneLLM to the `Microsoft.Extensions.AI` abstractions (version 10.11 of `Microsoft.Extensions.AI.Abstractions`) in both directions. It references the OneLLM core package and that one package. | Direction | What | Entry point | |---|---|---| | OneLLM → Microsoft.Extensions.AI | A model as `IChatClient` | `runtime.AsChatClient("claude-sonnet")` | | | A model as `IEmbeddingGenerator>` | `runtime.AsEmbeddingGenerator(model)` | | | A model as `ISpeechToTextClient`, `ITextToSpeechClient`, `IImageGenerator` | `runtime.AsSpeechToTextClient(model)`, `AsTextToSpeechClient`, `AsImageGenerator` | | | A `OneLlm` tool as an `AIFunction` | `tool.AsAIFunction(timeProvider)` | | Microsoft.Extensions.AI → OneLLM | Any `IChatClient` (and an optional embedding generator) as a provider | `runtime.Providers.AddChatClient("name", client)` | | | An `AIFunction` as a tool | `function.AsTool()` | The speech, text-to-speech and image abstractions are marked experimental by Microsoft.Extensions.AI (`MEAI001`). The bridge opts in inside one source file; code that uses `OneLlmSpeechToTextClient`, `OneLlmTextToSpeechClient` or `OneLlmImageGenerator` sees the same diagnostic on the Microsoft.Extensions.AI types it touches. ## A OneLLM model as a client ```csharp var rt = new Runtime(); IChatClient client = rt.AsChatClient("claude-sonnet"); ChatResponse r = await client.GetResponseAsync("Summarize this", new ChatOptions { MaxOutputTokens = 500 }); decimal dollars = (decimal)r.AdditionalProperties![OneLlmProperties.CostTotal]!; ``` The model resolves through the runtime's catalog, so requests are adjusted to what the model accepts (a parameter it rejects is dropped and reported in `OneLlmProperties.Warnings`) and every call is priced. `ChatOptions.ModelId` names another catalog model for one request. The client does not run functions: wrap it with Microsoft.Extensions.AI's function-invoking middleware, or use a OneLLM agent. Over a bare provider the client sends the model name as given and prices nothing: `new OneLlmChatClient(provider, "model-name", timeProvider)`; pass a `ResolvedModel` instead of the name to adjust and price. ### Options | `ChatOptions` | OneLLM `ChatRequest` | |---|---| | `Instructions`, `System`/`Developer` messages | `System` (joined with a blank line, instructions first) | | `MaxOutputTokens`, `Temperature`, `TopP`, `StopSequences` | the same fields | | `Tools` (`AIFunctionDeclaration`) | `Tools`; the declaration's `strictJsonSchema` property sets `Strict` | | `Tools` (`HostedWebSearchTool`, `HostedCodeInterpreterTool`) | `HostedTools` `web_search`, `code_interpreter` | | `ToolMode` | `ToolChoice` (`Auto`, `None`, `Required`, or the named tool) | | `AllowMultipleToolCalls` | `ParallelToolCalls` | | `ResponseFormat` | `ResponseFormat`: text, any JSON, or the JSON schema with its name | | `Reasoning.Effort` | `Reasoning.Effort` (`none`, `low`, `medium`, `high`, `xhigh`) | | `Reasoning.Output` | `Reasoning.Summary`: `Summary` is `auto`, `Full` is `detailed`. Another spelling goes through `onellm.extraParameters`. | | `AdditionalProperties["onellm.cache"]`, `["onellm.serviceTier"]`, `["onellm.userId"]`, `["onellm.extraParameters"]` | `Cache`, `ServiceTier`, `UserId`, `ExtraParameters` | | `Seed`, `TopK`, `FrequencyPenalty`, `PresencePenalty`, `ConversationId` | not carried; the first four are named in a warning | ### What comes back Text, tool calls and reasoning arrive as `TextContent`, `FunctionCallContent` and `TextReasoningContent` in the order the model produced them, so the order of parts inside a turn is the order of the `Contents` list. Streaming sends text and reasoning as deltas; a tool call arrives complete, once its arguments have finished streaming. A stream ends with one update that carries `FinishReason` and one `UsageContent`, so Microsoft.Extensions.AI's `ToChatResponse()` counts usage once. | OneLLM | Microsoft.Extensions.AI | |---|---| | `Usage.TotalInputTokens` (new + cache read + cache write) | `UsageDetails.InputTokenCount` | | `Usage.CacheReadInputTokens` | `CachedInputTokenCount` | | `Usage.CacheCreationInputTokens` | `AdditionalCounts["onellm.cacheCreationInputTokens"]` | | `Usage.Counters["reasoningTokens"]` | `ReasoningTokenCount` | | `Usage.OutputTokens` | `OutputTokenCount` | | `FinishReason` `Stop`, `Length`, `ToolCalls`, `ContentFilter` | the matching `ChatFinishReason`; `Refusal` and `PauseTurn` become `refusal` and `pause_turn` | | `TextPart` annotations | `CitationAnnotation` (title, URL, snippet, span; the original `Annotation` is `RawRepresentation`) | | `AssetPart` with inline bytes / a public URL | `DataContent` / `UriContent`; a stored asset with neither is kept whole in `OneLlmPartContent` | | `FileRefPart` | `HostedFileContent` | | `JsonPart` | `TextContent` holding the JSON text | | `RefusalPart` | `TextContent` with `onellm.refusal = true` | | `ReasoningPart` | `TextReasoningContent`: `Text` is the display text, `ProtectedData` the provider's opaque JSON, `onellm.provider` names the provider | | hosted tool calls, provider parts | `OneLlmPartContent` carrying the `Part` | ### Keys in `AdditionalProperties` On the `ChatResponse` (and on the final `UsageContent` of a stream), and on an embedding result: | Key | Value | |---|---| | `onellm.cost` | the call's `OneLlm.Messages.Cost`: line items, each with its counter, quantity, unit price and amount, and the total | | `onellm.cost.total` | the total as a `decimal`, in US dollars | | `onellm.warnings` | `string[]`: stripped parameters, a missing price, options the bridge could not carry | Cost is present only when the model is priced in the catalog. Money is always `decimal`. The lossy edges, carried so they survive a round trip through Microsoft.Extensions.AI and back: | Key | On | Meaning | |---|---|---| | `onellm.when` | `ChatMessage`, `FunctionCallContent` | the entry's `when` expression | | `onellm.injected`, `onellm.injectedBy` | `FunctionCallContent` | the call was injected by the runtime rather than asked for by the model, and by what | | `onellm.routine` | `FunctionCallContent` | the tool source reference the call resolved to | | `onellm.invalidInputs` | `FunctionCallContent` | the raw argument text when the model's arguments were not a JSON object (`Arguments` is then `null`) | | `onellm.isError`, `onellm.resultParts` | `FunctionResultContent` | the tool reported an error; the original result parts, kept in process | The speech, text-to-speech and image types carry usage on their `Usage` property; speech-to-text and text-to-speech responses also carry the cost keys. `ImageGenerationResponse` has no property dictionary, so an image call's cost is not on it: use the runtime's result. ## A client as a provider ```csharp var rt = new Runtime { Catalog = ModelCatalog.Create([CatalogLayer.FromFile(CatalogLayer.User, "my-models.yaml")]) }; rt.Providers.AddChatClient("acme", someChatClient, someEmbeddingGenerator); // someEmbeddingGenerator is optional AskResult a = await rt.AskAsync(new AskRequest { Model = "acme-large", Prompt = "hi" }); // priced from my-models.yaml ``` The catalog entry supplies what the client cannot: the model's name patterns, context window, capabilities and prices. Its `name` must be the provider name passed to `AddChatClient`. The client's usage becomes OneLLM `Usage`: `InputTokenCount` is the whole input and `CachedInputTokenCount` the part read from a cache, so the cached tokens are priced at the cache-read rate and not twice. Tool definitions go out as declarations, results come back as function-result content, and a streamed update list becomes OneLLM stream events that build the same `Message` a non-streaming call builds. ## Tools `tool.AsAIFunction(time)` passes the tool's name, description, input schema and output schema through unchanged, so `function.JsonSchema` equals `tool.Definition.InputSchema`. The function runs the tool in a context with **no capabilities**; pass a `Func` to grant the files, shell or network the tool needs. A tool that reports an error makes the function throw `InvalidOperationException` with the tool's message, which Microsoft.Extensions.AI's function-invoking client reports to the model. `function.AsTool()` does the reverse: the schema is read from `JsonSchema`, an `ApprovalRequiredAIFunction` becomes a tool with `RequiresApproval`, and a result becomes text (a string), a JSON part (a JSON value) or parts (a content list). The name must be a valid tool name (letters, digits, `_`, `-`, at most 64): a function named `has.dot` is refused with `ArgumentException`. Tools made with `AIFunctionFactory.Create` use reflection and are not trim or AOT safe; that is a property of Microsoft.Extensions.AI, not of the bridge. Declarations, `AsAIFunction` and `AsTool` over a hand-written or generated `ITool` are. ## Reads like A table for readers who know the Microsoft.Extensions.AI calls. It maps calls in the public documentation to OneLLM calls and makes no comparison. | Microsoft.Extensions.AI | OneLLM | |---|---| | `chatClient.GetResponseAsync(messages, options)` | `Llm.GenerateAsync(model, prompt, attachments)`; `Runtime.AskAsync(new AskRequest { … })`; `Llm.Chat(model)` for turns | | `chatClient.GetStreamingResponseAsync(...)` | `Llm.StreamAsync(model, prompt)`; `Llm.StreamTextAsync(model, prompt)` | | `GetResponseAsync` (structured output) | `Llm.AskAsync(model, prompt, typeInfo)` | | `ChatOptions.ModelId` | the model argument: a catalog name, alias or `provider:model` | | `ChatOptions.Tools` with `AIFunctionFactory.Create` | `[Tool]` methods (generated, AOT safe) or `Tool.Create(delegate)`; `ITool` | | `FunctionInvokingChatClient` | the agent runners and their tool loop | | `UsageDetails` | `Usage` and `Cost` (line items) on every result | | `IEmbeddingGenerator>.GenerateAsync` | `Llm.EmbedAsync(model, text)` | | `ISpeechToTextClient.GetTextAsync` | the transcription provider capability (`ISpeechToTextProvider`) | | `ITextToSpeechClient.GetAudioAsync` | the speech provider capability (`ITextToSpeechProvider`) | | `IImageGenerator.GenerateAsync` | the image provider capability (`IImageProvider`) | | `ChatClientBuilder` pipelines | `runtime.AsChatClient(model)` as the inner client of any pipeline | | `AIFunction` | `ITool` (`function.AsTool()`, `tool.AsAIFunction(time)`) | | `OpenTelemetryChatClient` | OneLLM emits GenAI telemetry itself; see `telemetry.md` | # Pricing, cost and token counting Normative notes for the catalog's price fields, how a `Usage` becomes a `Cost`, and how input tokens are counted. Money is `decimal` everywhere; nothing is rounded except when a cost is displayed (`Cost.Display`, half away from zero, 6 places by default). ## Cost lines A cost is a list of lines plus their sum. A line is one usage counter: `amount = quantity / per * unitPrice * multiplier`, with the price's `asOf` date and source URL on the line. | Counter | Price field | |---|---| | `inputTokens` (not cache reads or writes) | `inputPerMillion` | | `outputTokens` (reasoning included) | `outputPerMillion` | | `cacheCreationInputTokens` | `cacheCreationPerMillion` | | `cacheReadInputTokens` | `cacheReadPerMillion` | | any other counter (`images`, `audioSeconds`, `characters`, `toolCalls.web_search`, `containerHours`, `inputAudioTokens`, ...) | an entry in `items` | The media counters and which models use them are listed in `media.md`. An item gives `price` for `per` units (default 1,000,000 for counters ending in `Tokens`, otherwise 1). `when` narrows it to a case (for example `{size: 1024x1024, quality: high}`): it applies when every key matches the call's qualifiers; of several items for one counter the one naming the most keys wins. An item for one of the four main counters beats the flat field. A counter with a quantity and no price is left out of the cost and reported as a warning. A model with no price at all has an unknown cost, except on a loopback server (`localhost`, `127.0.0.0/8`, `::1`), where it is free unless the catalog says otherwise. ## Subset counters A counter that counts tokens already inside another is a subset: `reasoningTokens` and the output audio/image counters inside `outputTokens`; `inputAudioTokens`, `inputImageTokens`, `inputVideoTokens` inside `inputTokens`; `cacheCreation1hInputTokens` inside `cacheCreationInputTokens`. If the model prices the subset, its tokens are removed from the parent's quantity and charged at the subset's price, so nothing is charged twice. If it does not, they stay in the parent. A model can declare more relations with `pricing.subsetOf` (`child: parent`). ## Long context, tiers and multipliers - **Long context.** When uncached input plus cache writes plus cache reads is strictly above `longContextThreshold` (200,000 when unset) and `longContextInputPerMillion` is set, the whole request is priced at the long-context rates: input and output take their long-context prices (output keeps its normal price when no long-context output price is given), and cache prices are multiplied by long-context input price over normal input price. - **Tiers.** A tier applies when its basis (`totalInputTokens` unless `basis` names a counter) is strictly above `above`. Its `multipliers` factor the prices of the named counters, or every counter under `*`. Of several tiers that apply, the highest `above` wins; they do not stack. - **Factors on every line:** the service tier's `pricingMultiplier` (`standard`, `default` and `auto` carry none; an unknown tier prices at the standard rate and warns), `multipliers.batch` for batch calls, `multipliers.region:` for a region, and any other named multiplier the caller asks for. A missing name warns and is not applied. ## Staleness Prices older than 180 days past `asOf` are stale: a priced call warns when the caller supplies today's date, and `PriceFreshness.FindStale` lists them. ## Request adjustment `RequestAdjuster` reconciles a chat request with the catalog entry. It strips what is harmless to drop (temperature and top-p when `supportsTemperature` is false, reasoning settings for a model with no `thinking`, a thinking budget the model does not take, caching for a model without it) and clamps values (thinking budget to `minBudget`/`maxBudget`, `maxOutputTokens` to the model's maximum), returning a warning with a stable code (`parameter_stripped`, `parameter_clamped`) for each. It refuses with `UnsupportedParameterException` what would change the request's meaning: tools or a forced tool for a model without tool support, a JSON schema for a model without structured output, an image, PDF, audio or video the model cannot read, a reasoning effort or service tier the model lacks, and chat for a decisions-only provider. A field the catalog leaves out means unknown and never strips or refuses. ## Token counting Three tiers, tried in order; every count says which one answered. 1. **Provider** – the provider's own count endpoint, where the provider has one. 2. **Tokenizer** – a local tokenizer built from the vendor's published vocabulary data, supplied through `ILocalTokenizer`. Text is counted exactly; message, tool and file allowances are the estimate's. 3. **Estimate** – characters per token by family (4.0 for OpenAI-style and Gemini models, 3.5 otherwise; overridable per model with the property `charsPerToken`), two-byte characters at double weight, CJK and other characters at one token each; 4 tokens per message and 3 for the reply; 8 per tool, and 346 once for Anthropic when any tool is present; images by the vendor's documented formula (Anthropic width × height ÷ 750 capped at 1,600; OpenAI 85 + 170 per 512 px tile after scaling to fit 2048 px and the short side to 768 px; Gemini 258 up to 384 × 384 and 258 per 768 px tile above); PDFs by page estimate; audio and video by duration estimate. The estimate errs high. Context usage is the last turn's total input tokens (uncached, cache writes and cache reads) divided by the model's `contextWindow`. ## Estimates and spend limits `CostEstimator` gives the worst case before sending: every input token at the dearer of the input and cache-write prices (when the request may write the cache) plus the full output allowance (the request's limit, otherwise the model's maximum), with long-context rates when the input is that large. `SpendGuard` enforces a per-call limit and a spend cap on those estimates; it holds reservations for calls in flight, so concurrent calls cannot pass the cap together, and refuses (rather than ignores) a limit it cannot check because the price or the output limit is unknown. ## Result cache `ResultCache` is an opt-in in-memory cache of chat answers keyed by a SHA-256 over everything that can change the answer. A hit costs nothing and sends nothing. # Instance stores and record sinks An instance store keeps instances between runs; a record sink keeps the audit trail of what a run did. OneLLM ships four stores and four sinks. Instance state is the truth; records are the trail of how it got there. | Store | Package | Keeps | Use it when | |---|---|---|---| | `FileInstanceStore` | `RecursiveAI.OneLlm` | one `.yaml` per instance in a directory, written by temp file and rename | The default for tools and agents on one machine. The files are readable, diffable and portable. | | `MemoryInstanceStore` | `RecursiveAI.OneLlm` | the instance text in the process | Tests, one-shot scripts, and runs that must leave nothing behind. | | `OneLiteInstanceStore` | `RecursiveAI.OneLlm.OneLite` | one row per instance in a OneLite (SQLite-format) file | You want many instances in one file, queries, or several processes sharing a store; trimming and NativeAOT matter. | | `EfInstanceStore` | `RecursiveAI.OneLlm.EntityFrameworkCore` | one row per instance in your own `DbContext` | Your application already uses Entity Framework Core and a server database. | | Sink | Package | Keeps | |---|---|---| | `JsonlRecordSink` | `RecursiveAI.OneLlm` | one `.jsonl` per instance, appended | | `EventRecordSink` | `RecursiveAI.OneLlm` | nothing; raises an event per record | | `OneLiteRecordSink` | `RecursiveAI.OneLlm.OneLite` | one row per record | | `EfRecordSink` | `RecursiveAI.OneLlm.EntityFrameworkCore` | one row per record in your `DbContext` | Several sinks can run at once through `RecordSinks`. A sink that throws is skipped for that record and reported to `RecordSinks.OnError`; it never fails a run. ## What every store does Every store passes the same conformance suite (`tests/OneLlm.Stores.Tests`): - `SaveAsync` replaces the stored instance atomically; a reader sees the old instance or the new one. A save that is cancelled before it starts leaves the old one. - `LoadAsync` returns an instance whose file text equals what was saved, unknown fields included; `null` for an unknown id. - `ListAsync` returns summaries newest first and applies `InstanceQuery` (state, caller, name contains, modified since, limit). - `AcquireAsync` hands out one lease per instance. A second taker, in the same process or (for file, OneLite and EF) another process or store handle on the same storage, gets `InstanceLockedException`. Disposing the lease frees it. - Ids are 1 to 128 letters, digits, `-` and `_` (`InstanceIds.Validate`). - `DeleteAsync` removes the instance; the audit trail stays. ## Leases that survive crashes The file store holds a lock file open. The database stores cannot tie a lease to a process, so a lease is a row with an expiry. The holder renews it every third of the lease duration (default 60 seconds, a constructor argument); a process that dies stops renewing, and the next taker may take the row once it has lapsed. The expiry uses the store's `TimeProvider`. ## OneLite ```csharp using RecursiveAI.OneLite; using OneLlm.OneLite; using Database db = await OneLiteStorage.OpenAsync("onellm.db"); // creates the file and the schema var store = await OneLiteInstanceStore.CreateAsync(db); var sink = await OneLiteRecordSink.CreateAsync(db); // the same database can hold both decimal spent = await sink.CostAsync("inst1", includeSubtree: true); var entries = await sink.ReadAsync("inst1", kind: "toolCall"); ``` The schema is versioned with `PRAGMA user_version` and created on first open; later versions append migration scripts, and a file from a newer package is refused. Times are stored as UTC ticks and costs as decimal text, so nothing depends on float or date formats. Each save is one write transaction. The adapter is trim and NativeAOT compatible (row mapping is generated at build time). ## Entity Framework Core ```csharp public class AppDb(DbContextOptions options) : DbContext(options) { protected override void OnModelCreating(ModelBuilder b) => b.ApplyOneLlm(); // tablePrefix: "OneLlm", schema: null } var factory = /* IDbContextFactory */; var store = new EfInstanceStore(() => factory.CreateDbContext()); var sink = new EfRecordSink(() => factory.CreateDbContext()); ``` `ApplyOneLlm` maps three tables (`OneLlmInstances`, `OneLlmRecords`, `OneLlmLeases`) into your model; the table names take a prefix and an optional schema. Create them with your own migrations (`dotnet ef migrations add`) or `EnsureCreated`; the package ships no migrations and no database provider. Each call uses a fresh short-lived context from your factory. - The instance row's `Version` is a concurrency token, bumped on every save. Writers that race on one row retry against the new row, so the last save wins, the same rule as the file store. - Times are stored as UTC ticks so they sort the same on every provider. Costs are `decimal(28,12)` and summed in the application, because some providers cannot aggregate decimals in SQL. - Entity Framework Core builds its model by reflection, so this package is not trim or NativeAOT compatible. Use the OneLite package where that matters. ## Costs and records Both database sinks store each record as its JSON line (the same text `JsonlRecordSink` writes, read back by `RecordReader.ParseLine`) plus columns for instance, kind, time, message and cost. `CostAsync(instanceId, includeSubtree)` sums inference and tool-call costs as `decimal`. A subtree is every instance reached by following `callerInstanceId` and `spawnerInstanceId` downward. Only those two kinds are summed; a compaction record's own cost field is not added, so a summarising call is counted once, as the inference record it produced. # Telemetry OneLLM emits OpenTelemetry traces and metrics for model calls through `System.Diagnostics`. Subscribe to the activity source and the meter named `OneLlm`; with an OpenTelemetry SDK that is `AddSource("OneLlm")` and `AddMeter("OneLlm")`. When nothing is subscribed, a call costs one check and allocates nothing. Names follow the OpenTelemetry GenAI semantic conventions (attribute registry: `https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/`). The conventions are still evolving; the names below are the ones OneLLM emits. ## Spans One client span per model call, named `{gen_ai.operation.name} {gen_ai.request.model}`, kind `Client`. | Attribute | Value | |---|---| | `gen_ai.operation.name` | `chat`, `embeddings`, `text_completion`, `execute_tool`, `invoke_agent`, ... | | `gen_ai.provider.name` | `anthropic`, `openai`, `gcp.gemini`, ... | | `gen_ai.request.model` | The model that was asked for. | | `gen_ai.response.model` | The model that answered. | | `gen_ai.response.id` | The provider's response id, when it gave one. | | `gen_ai.response.finish_reasons` | One-element array. | | `gen_ai.usage.input_tokens` | All input tokens, including cache reads and writes. | | `gen_ai.usage.output_tokens` | Output tokens, including reasoning. | | `gen_ai.usage.cache_read.input_tokens` | When not zero. | | `gen_ai.usage.cache_creation.input_tokens` | When not zero. | | `gen_ai.usage.reasoning.output_tokens` | When not zero. | | `error.type` | The exception type, on failure; the span status is `Error`. | Prompts, completions and tool arguments are not recorded on spans. ## Metrics | Instrument | Kind | Unit | Tags | |---|---|---|---| | `gen_ai.client.token.usage` | histogram of long | `{token}` | operation, provider, request model, `gen_ai.token.type` = `input` or `output` | | `gen_ai.client.operation.duration` | histogram of double | `s` | operation, provider, request model, `error.type` on failure | # Errors and retries ## Errors Every failure OneLLM raises derives from `OneLlmException`. Provider failures derive from `ProviderException` (provider name, HTTP status). API keys are held in `Secret` and removed from every message. | Type | When | |---|---| | `AuthException` | 401, 403: key missing, wrong or not allowed. | | `RateLimitException` | 429; carries `RetryAfter`. | | `OverloadedException` | 503, 529; carries `RetryAfter`. | | `ContextOverflowException` | The request is larger than the model's context window. | | `ContentFilterException` | The provider's safety filter blocked the request or response. | | `SpendLimitReachedException` | The account's own spending limit at the provider was reached (Anthropic: a 400 whose message starts "You have reached your specified API usage limits"). Never retried. | | `BadRequestException` | 400, 404, 409, 413, 422: the request is wrong. Never retried. | | `TransportException` | The connection failed or timed out. | | `MalformedResponseException` | The response is not what the API documents. | | `RefusedException` | The model declined to answer. | | `BudgetExceededException` | A spend or token cap was reached (`BudgetKind`). | | `CapabilityMissingException` | A feature needs a package or grant that is not present; the message names it. | | `BadDefinitionException` | A definition file is invalid; carries file, line and column. | | `ModelNotFoundException` | A model name resolves to nothing. | Codecs map statuses with `ProviderErrors.FromStatus` and then look in the body for causes only the provider reports (a context overflow arriving as a plain 400). ## Retries `RetryPolicy` (defaults: 2 retries, base delay 500 ms, cap 30 s, jitter 25%, longest accepted `Retry-After` 60 s). - **Retried:** 408, 429, 500, 502, 503, 504, 529 and transport failures. - **Never retried:** other statuses, including every 4xx that says the request is wrong, and a 429 that reports an exhausted quota rather than a rate. - **Delay:** the backoff doubles per attempt from the base delay, up to the cap, plus 0–25% random jitter; when the provider's `Retry-After` (seconds or an HTTP date, or `retry-after-ms`) is longer, that is used. A `Retry-After` beyond the accepted maximum is not waited out: the error is returned at once. - **Timeouts** apply per attempt, up to the response headers; waits use the runtime's `TimeProvider`, so tests run on a fake clock. - **Cancellation** ends a request or a wait immediately with `OperationCanceledException`. ## HTTP All clients share one `SocketsHttpHandler` with `AllowAutoRedirect = false`: a redirect could carry credentials to another host, so a redirect response is an error instead. # The `onellm` command line `onellm` runs OneLLM definitions from a terminal. It is a .NET tool (`RecursiveAI.OneLlm.Cli`, command `onellm`; it hosts the core, `.Web` and `.JavaScript` packages) built on the same `Runtime` the library exposes, and it is line-oriented: no full-screen interface, results on standard output, progress on standard error, so it composes in pipes and scripts. ``` dotnet tool install -g RecursiveAI.OneLlm.Cli onellm --help ``` API keys are read from the environment only (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, `BRAVE_API_KEY` for web search, and the others the catalog names). There is no option to pass a key. ## Commands | Command | What it does | |---|---| | `onellm run [options]` | Runs a definition. The file decides the mode (below). | | `onellm chat [--system text]` | Talks to a model: reads a line from standard input, streams the answer, repeats. `/exit` or end of input stops. The chat is saved as an instance. | | `onellm instances list [--state-filter s] [--name text] [--limit n]` | Lists saved instances, tab-separated: id, state, modified (UTC), name, model. | | `onellm instances show ` | Prints an instance: header, cost, and each message with its tool calls and their results (clipped). | | `onellm instances resume [--prompt text]` | Continues an instance that is waiting for a message or for approvals. Takes the same grant flags as `run`. | | `onellm instances fork [--at messageId]` | Copies an instance (up to a message) into a new one and prints its id. | | `onellm cost [id]` | With an id: cost by model, for the instance and for it with its sub-agents. Without: everything in the state directory. `--since 24h`, `7d`, `2w` or a date limits it. | | `onellm catalog [model]` | Without a model: every catalog model and its context window. With one: each value and, in brackets, the layer that set it (`shipped`, `user`, `code`). | | `onellm examples [show ]` | Lists the example agents built into the tool, or prints one. | ### Modes of `run` | Target | Mode | |---|---| | `examples/` | The embedded example agent of that name | | `*.yaml` (and a directory holding `agent.yaml`) | Agent: the tool loop | | `*.flow` | Scripted: the prompts in order, one model call each, no tools | | a directory holding `decider.yaml`, or that file | Decider: asks its questions, prints its outputs as JSON. A `decider.js` beside it supplies the logic. | Options of `run`: - `--input k=v` (repeatable) sets an input; `k=@file` reads the value from a file. `--input-json '{...}'` (or `@file`) sets several, with types. - `--prompt text` is an agent's first user message (agents only). `--model name` replaces the definition's model. - `--max-cost usd` stops a run that would spend more. - `--tools file-or-folder` (repeatable, every command that runs agents) loads JavaScript tool files (`.js`, with an optional `.tool.yaml`) so definitions can list them by name under `references`. A tool file is code you chose to load; what it may touch is still decided by the grants. - `--yes` approves every held tool call without asking. `--quiet` hides the tool-call lines on standard error. - Common to every command: `--root dir` (default: the current directory) and `--state dir` (below). The result is printed to standard output: an agent's `task_complete` result (its summary goes to standard error), the typed output when the definition declares one, otherwise the last answer. The last line on standard error says the instance id, its state, turns and cost (US dollars, six places, halves away from zero). ## Grants Definitions never grant anything. The tool grants only what its flags say: | Flag | Grants | |---|---| | (none) | Reading and writing files under `--root`; nothing else. No shell. No network for tools. | | `--allow-shell` | Any shell command | | `--allow-shell=git,dotnet` | Only these programs, run without a shell (arguments are never interpreted) | | `--allow-shell=ask` | Any command, each one confirmed at the terminal | | `--allow-net` | Public hosts (loopback, private and link-local addresses stay refused) | | `--allow-net=a.example.com,*.b.org` | Only these hosts | A definition that references a tool the grants do not cover (a shell tool without `--allow-shell`, `http_request` without `--allow-net`) does not load: the run stops with exit code 3 naming the tool and the missing grant, before any model call. Sub-agents hold at most what their parent holds. When a tool call needs approval (a tool marked `requiresApproval`, or a hook asking), the terminal asks `Approve ? [y/N]`. The default is no, and so is end of input. The held call is saved with the instance, so a run can also be left and answered later with `onellm instances resume `. ## JavaScript The tool includes the `.JavaScript` package. `run_javascript` is always offered to agents that list it (the code is sandboxed and can use only the grants of the run). Hook files listed in a definition's `hooks:` run as interceptors and observers, and a `decider.js` next to a `decider.yaml` is the decider's logic. See [JavaScript tools and hooks](https://recursiveai.org/onellm/docs/javascript/). ## Where things are kept Under `/.onellm/` unless `--state` says otherwise: - `instances/`: one YAML file per instance, written atomically. Agents can read it (it is under the root), which lets them look at other runs; point `--state` outside the root to prevent that. - `records/`: the JSON-lines audit records (runs, messages, inference calls, tool calls, errors). ## Exit codes | Code | Meaning | |---|---| | 0 | Done. Also: the agent answered and waits for your next message (the last line says how to resume). | | 1 | The run failed: a provider error after retries, a spend cap, a hook block, an instance that ended in `error`, an unknown instance id. | | 2 | The command line is wrong. | | 3 | A file, definition, grant, model or key problem: a missing file, an invalid definition (with file, line and column), an ungranted reference, an unknown model, a missing API key. | | 4 | The agent used all its iterations without finishing. | | 130 | Cancelled with Ctrl-C. The partial run is kept as a stopped instance. | ## The example agents `onellm examples` lists the eight embedded agents; `agents/README.md` in the repository describes each with its grants and a representative task. Sub-agents of the orchestrator may start any of them by name (`examples/`). ## Notes - The tool is a managed .NET tool, not published as NativeAOT; the library it hosts is trim- and AOT-safe. - `onellm catalog` shows the effective catalog: set `ONELLM_CATALOG` to a file or directory of `llm-provider.yaml` files to add a user layer. - `decider.ts` runs when it uses only erasable TypeScript (see the JavaScript page). - Web search (`search_web`) is available to agents when `BRAVE_API_KEY` is set; fetching pages (`fetch_webpage_markdown`) needs `--allow-net`. # Example agents Eight agents ship inside the `onellm` tool. Run one by name, or copy it with `onellm examples show ` and change it. ``` onellm run examples/ --prompt "" [grant flags] ``` Each is a single `agent.yaml` in its own folder here. They use the default model `claude-sonnet`; pass `--model` to use another. Nothing is granted by the definition: files under the working directory are always available, and the shell is available only where the table says so. An example that needs a shell does not load without the flag, and says why. | Example | What it does | Grants | Representative task | |---|---|---|---| | `cs-tool-writer` | Writes a C# `[Tool]` class, builds it, says how to register it | `--allow-shell=dotnet,git` | `--prompt "In ./src/Tools, write a tool that converts a length between metres, feet and miles; add a test."` | | `js-tool-writer` | Writes a sandboxed JavaScript tool and its `.tool.yaml` | none | `--prompt "In ./tools, write a JS tool that reshapes {firstName, lastName} records into {name} records."` | | `provider-writer` | Adds a provider: a catalog preset for an OpenAI-compatible server, or a provider class for a native API | `--allow-shell=dotnet,git`; `--allow-net=` lets it read vendor documentation | `--prompt "Add a catalog preset for the OpenAI-compatible server at https://api.example.test/v1; its docs are at https://docs.example.test."` | | `agent-writer` | Designs an agent definition from a description of the job | none | `--prompt "Write ./agents/changelog.yaml: an agent that reads the git log text in ./log.txt and writes CHANGELOG.md."` | | `script-writer` | Writes a scripted `.flow` | none | `--prompt "Write ./commit-message.flow: summarize a diff, then tighten it to a 72-character subject plus a body."` | | `decider-writer` | Writes a `decider.yaml`, exercises it with `run_decider` | none (the decider's model needs its key in the environment) | `--prompt "Write ./deciders/triage: classify a support ticket by area, urgency, and whether the customer is angry."` | | `hook-writer` | Writes a C# hook with tests | `--allow-shell=dotnet,git` | `--prompt "Write a BeforeToolCall hook that blocks write_file outside ./out, with tests."` | | `orchestrator` | Splits a job, starts the other examples (or any definition file) as sub-agents, watches them, combines the results | whatever its sub-agents need: sub-agents hold at most what the orchestrator holds | `--prompt "Write a tool, a flow and a decider for ticket triage under ./triage, each by its own agent."` with `--allow-shell=dotnet,git` | ## What counts as done Each agent ends by calling `task_complete` with the files it wrote and what it could not verify. A run is a success when: - the files named in the task exist where the task said, - the agent's own checks passed (the build and tests for `cs-tool-writer`, `hook-writer` and `provider-writer`; `run_decider` results for `decider-writer`), - the `task_complete` result names anything it did not check. The scripted end-to-end tests in `tests/OneLlm.Cli.Tests` run every example through the real tool loop against a scripted model, so the definitions, references and grants are checked on every build. Runs against live models are a release check. # Definitions, flows and deciders: format v1 This is the file format OneLLM reads and writes for agent definitions, scripted flows and deciders. It is YAML 1.2. The format is versioned as a whole; this is version 1. OneLLM reads every key described here and keeps every key it does not know. ## Rules that apply to every file - **Unknown keys are kept.** A key this document does not describe is loaded into the type's `ExtensionData` and written back with the same value. This holds at every level: the definition, a tool reference, a hook entry, a message, a message part, a tool call, a decider question. - **A key with a null value and an absent key mean the same.** OneLLM does not write keys whose value is null. - **A list that was written as an empty list stays an empty list** when the file is saved again; an absent list stays absent. - **Saving something that was not changed returns the loaded text byte for byte**, comments included. Once a value is changed the file is written again from the model: the same keys and values, with unknown keys kept, but not the comments or the original layout. - **Loading never grants a capability.** `references` name tool sources the agent would like; the runtime's grants decide. - **Errors carry a position.** A malformed file, a value of the wrong type, or a refused reference fails with the file name and the 1-based line and column of the node (`BadDefinitionException`). - **Strict is the default.** Strict loading fails on an ungranted reference, an invalid `missingKey` or a hook without a `file`. Lenient loading drops the offending entry and records the reason in the definition's `Warnings`. ## Agent definition (`agent.yaml`, `*.flow`) A mapping. Every key is optional. | Key | Type | Meaning | |---|---|---| | `name` | string | Display name. | | `description` | string | One line. | | `model` | string | A catalog model name or alias, or `provider:model`. | | `serviceTier` | string | Service tier to request (`flex`, `priority`, ...). | | `thinking` | mapping | `effort` (string) and/or `budget` (integer): request-side reasoning controls. | | `temperature` | number | Sampling temperature. | | `maxIterations` | integer | The most model turns one run may take. | | `maxTokens` | integer | The most output tokens per turn. | | `systemPrompt` | string | A template. Rendered when a run starts and again each turn. | | `references` | list | Tool sources, below. | | `workingDirectory` | string | A template. | | `initialPrompts` | list | Messages that open every run, below. | | `inputs` | mapping | Default input values. Run inputs are merged over them. | | `headers` | mapping of string to string | Extra HTTP headers for HTTP tool sources; values are templates. | | `invocationMode` | string | `interactive`, `delegated` or `autonomous`. | | `llmTurnLogs` | boolean | Keep raw request and response logs for each turn. | | `missingKey` | string | `default` or `error`. What a template does with a key it cannot find. `default` renders ``. Agents and scripted runs default to `default`; deciders are always `error`. | | `hooks` | list | Hook files, below. | ### `references` Each entry is a mapping with one of: - `routine`: a tool source path or URI, with optional `tools` (list of tool names; absent means all of them, an empty list means none) and `required` (boolean); - `swarm`: a group of sources to take every tool from. Other keys are kept. An entry that names a source the runtime has not granted is refused (strict) or dropped (lenient). ### `initialPrompts` A list of messages in the message format (`spec/messages.md`): `user` and `assistant` lists of parts, and `toolCalls`. Two keys matter only here: - `when` (on a message or on a tool call): a template. The message or call runs unless the rendered text, trimmed and lower-cased, is empty, `false`, `0`, `no` or `off`. - tool calls without a `result` are run before the first turn; ones with a `result` are injected as they are. A `user` part may be `from: ` to expand a list from the template data (for example `inputs.images`) into several parts. Messages in `initialPrompts` usually have no `id`; the runtime gives them one when they enter an instance. ### `contextStrategy` Optional. Keeps a long run inside the model's window. `contextStrategy: {type, triggerTokens | triggerPercent, keepLast}` or just the type name. It runs at a turn boundary, when the previous model call's context (input plus output tokens) reached the trigger (`triggerTokens`, or `triggerPercent` of the model's window, default 70): - `clearToolResults`: the results of all but the latest `keepLast` (default 3) tool-calling turns are replaced by the note `[tool result cleared to save context]`. Every call keeps a result, so no call is orphaned, and messages before the first change are untouched, so a provider's cached prefix stays valid up to there. Doing it again changes nothing. - `compact`: messages before a user message at least `keepLast` (default 6) messages from the end are summarized by a model call and replaced by one user message that starts with `[Summary of the earlier conversation]`. The summarizer is the shipped `compact` flow, replaceable on the `Runtime`; it runs as an instance of its own below this one, so its cost is in the subtree. - `providerNative`: the provider compacts the conversation, when it can. A provider that cannot is skipped with a warning. Each run adds an entry to `compactions` (`spec/instances.md`). A host hook can veto one. ### `hooks` A list of mappings: `file` (required, relative to the definition) and `points` (list of hook point names; absent means every point the file handles). Host hooks registered on the runtime wrap these. ## Flows (`*.flow`, `flow.yaml`) A `.flow` file has the shape of an agent definition and is run in the scripted mode by default: the `initialPrompts` run in order, one model call per user prompt, no tools. `references` in a flow are ignored with a warning. `flow.yaml` beside it is metadata: `name`, `description`, `version` and any other keys (version history, input examples, rubrics). OneLLM keeps the whole file and does not interpret the other keys; they are available as `FlowMetadata.Other`. ## Decider (`decider.yaml`) A mapping with: | Key | Type | Meaning | |---|---|---| | `model` | string | Required. The engine or model to ask. A catalog model of kind `decision` is a decisions engine; any other model answers as a chat model. | | `state` | any | A template tree rendered with the run's inputs. It is what the questions are about. | | `questions` | mapping | Required, at least one. Question id to question. Ids are 1 to 64 characters of letters, digits, `_`, `-`, `.`. | | `llm` | mapping | Optional. For chat-model engines only: `samples` (1 to 100, default 1), `temperature` (0 to 2), `thinking` (an effort name, or `{effort, budget}`), `serviceTier`. | | `cache` | mapping | Optional. `key` (1 to 64 characters; default the decider's name) partitions prompt-cache reuse; `explicit: true` marks the end of the state as the cached prefix. Engines that cannot cache ignore it. | Other top-level keys are kept as extension data and written back unchanged. A question is a mapping with: - `type`: `choice`, `score` or `noul` (a yes/no judgment with a probability); - `instructions`: required; a template string, or a mapping of named parts, each a template; - `criteria`: for `choice`, a mapping from value to description with 2 to 255 entries; for `score`, the levels from lowest to highest as a list (or a mapping from label to description) with 2 to 10 entries; for `noul`, optional conditions (`true` and `false` keys that say what counts as yes and as no); - `forEach`: exactly one `{{ ... }}` action that gives a list. The question is asked once per element, with `.item` and `.index` in scope. Its answer is a list in element order; an empty list asks nothing. ### Templates in a decider Every string that contains `{{` is a template. Missing names are always errors (`missingKey: error`), `embed` is not available, and code regions are not escaped. A string that is exactly one action, such as `"{{ .inputs.customer }}"`, is a *whole-value* template: the action's value is placed in the JSON as structured data (an object stays an object, nothing becomes `null`). Any other string is rendered as text. ### Request encoding Structured values are written as compact JSON in the form Go's `encoding/json` produces, because engines and caches see the bytes: object keys sorted, `<`, `>` and `&` written as `\u003c`, `\u003e` and `\u0026`, U+2028 and U+2029 escaped, and numbers read as 64-bit floats and printed in their shortest form (plain decimal between 1e-6 and 1e21, exponent form outside). ### Runs and answers A run is an instance named after the decider. Its state is `stopped` when answered and `error` otherwise; the extension key `decider` holds the engine, the origin (`api`, `agent`, `playground`, `replay`, `eval`; the last two kinds stay out of statistics), the models, the calls with their cost and the answers. Answers, by question id (a repeated question gives a list), each with `type`: | Type | Fields | |---|---| | `choice` | `choice` (the winning key), `confidence` (0 to 1), `probabilities` (key to probability) | | `score` | `score` (the expected level, 0 for the lowest), `confidence`, `probabilities` (level to probability), `legend` | | `noul` | `noul` (the probability of yes) | Logic is optional and separate from the file: `prepare` turns the caller's inputs into the template inputs, `decide` turns the answers into the outputs. A decider with no logic returns its answers as its outputs. A `decider.schema.json` beside the definition may describe the inputs (`inputs`, with `inputsDeclared`); the keywords `type`, `properties`, `required`, `additionalProperties`, `items`, `enum`, `const`, `minimum`, `maximum` and `anyOf` are enforced before anything is sent, and all others are ignored. # Instances: format v1 An instance is one run of a definition and everything it has said and done. It is a YAML file named `.yaml`. The rules for unknown keys, null values, empty lists and byte-for-byte saving in `spec/definitions.md` apply here too. ## Layout An instance file is an agent definition (every key of `spec/definitions.md`, as they were when the run started) plus these keys: | Key | Type | Meaning | |---|---|---| | `instanceId` | string | 1 to 128 characters of letters, digits, `-` and `_`. It is the file name. | | `agentDefinition` | string | Path or URI of the definition the run started from. | | `requestedModel` | string | The model as first requested, before alias resolution. `model` holds the resolved name. | | `maxImageSize` | integer | The longest image edge in pixels the instance sends. Only ever tightened. | | `state` | string | The lifecycle state, below. | | `callerInstanceId` | string | The instance that last messaged this one. | | `spawnerInstanceId` | string | The instance that started this one. Set once. | | `callerToolCallId` | string | The tool call of the caller that started this instance. | | `delegationPrompt` | string | The prompt a calling agent gave when it started this instance. | | `runStartedAt` | string | RFC 3339 time the current run started. | | `metadata.renderedSystemPrompt` | string | The system prompt as rendered for the latest turn: what the model saw. | | `pendingApprovals` | list | Tool calls waiting for a decision, below. | | `compactions` | list | Records of context compaction, below. | | `messages` | list | The conversation, in the message format (`spec/messages.md`). | OneLLM writes the keys in this order, with `messages` last, so that a long conversation can be appended to without disturbing the header. A reader must accept any order. Keys other tools put in an instance file (a bound subscription, a wait record, context watchers, ...) are unknown keys and are kept. `metadata` keys other than `renderedSystemPrompt` are kept. ## States `state` is a string. The values OneLLM uses: | Value | Meaning | |---|---| | `running` | A run is in progress. | | `waiting` | The agent finished its turn and waits for input. | | `waiting_for_approval` | A tool call needs a person's decision; see `pendingApprovals`. | | `stopped` | The run ended. A new message continues the instance. | | `error` | The run failed. | Any other string loads and saves unchanged. A runner that finds a state it does not know treats the instance as stopped. ## `pendingApprovals` Each entry: `toolCallId`, `tool`, `messageId` (the message that holds the call, when it has been written), `inputs`, `reason` and `requestedAt` (RFC 3339). Approvals are saved with the instance, so an approval or denial still works after a restart. Unknown keys are kept. A decision is written into the same entry as two more keys: `decision` (`approved` or `denied`) and, for a denial, `reason` for the person's note. The entry stays until the run that follows has answered the call. While a turn waits for approval, its assistant message is saved with the held calls unanswered; the other calls of the turn already have their results. When every entry is decided, the run executes the approved calls, answers the denied ones with an error result that carries the reason, and replaces the saved message with the finished turn. ## `compactions` Each entry records one context compaction: `strategy` (`clearToolResults`, `compact` or `providerNative`), `at`, `replacedMessageIds`, `summaryMessageId` and `cost` (a decimal in USD, when the compaction made a model call). Two unknown keys are written: `atMessageId` (the model call whose context triggered it, so a strategy runs once per call) and `replacedCost` (a decimal as a string: what the removed messages and their tool calls cost, so the instance's total does not shrink when they go). ## Keys the agent loop adds Written as unknown keys, so other readers keep them: | Key | Meaning | |---|---| | `error` | `{type, message}` of the failure that put the instance in `error`. A run that reaches `maxIterations` ends in `error` with type `MaxIterations` and the message `max iterations (N) reached` (`maxIterations` defaults to 200). | | `cancelled` | `true` when the latest run was cancelled. The instance is `stopped` and keeps the partial turn: finished tool calls have their results, interrupted ones an error result saying they were cancelled. | | `capabilities` | The grants the run started with, so a resumed instance is bounded by them: `readRoots`, `writeRoots`, `workingDirectory`, `shell` `{enabled, approval, commands?}` and `network` `{anyPublic, hosts}`; and `tools`, the tool names a sub-agent started from a file was limited to. A reader that ignores it loses nothing but the narrowing. The runtime intersects it with its own grants and never widens from it. It is written with `capabilitiesRecorded: true`. A file with the marker but no well-formed `capabilities` entry is not run (it gets no grants); a file with neither, written before grants were recorded, runs with the resuming runtime's grants. | | `askUser` | `{toolCallId, messageId}` of an `ask_user` call that has no answer yet. The instance is `waiting`; the next message from the person or host (source `api` or `user`) becomes that call's result instead of a separate user message; other messages are added behind it. | ## Fork and duplicate A **fork** copies an instance under a new id, optionally only up to and including one message. It starts stopped with no pending approvals and no `runStartedAt`, and records where it came from in an unknown key `forkedFrom` (`instanceId`, `messageId`). A **duplicate** is a fork of the whole conversation. ## Assets Assets of an instance live beside it, in a directory `.assets/`, in a file store. Messages refer to them by `asset` (storage, key, hash, content type, size, name). ## Persistence layout and cost A save writes a temporary file in the same directory, flushes it to disk and renames it over the instance, so a reader sees the old file or the new one. A lock file `.lock`, held open without sharing for the life of a run, keeps a second runner out, across processes; the operating system drops it if the holder dies. Messages are immutable once appended. A save re-serializes the header and any message it has not written before, and reuses the text of the others, so adding a turn to a long instance costs the new turn plus the write of the file. # Message format (version 1) The normative description of how OneLLM represents and stores conversation messages. It covers the in-memory model, the file shape (YAML and JSON carry the same keys), the order index, unknown-field handling and how streamed events build a message. The file shape follows the lists used by the agent definition format OneLLM reads and writes (`user`, `assistant`, `toolCalls`, `reasoning`, ...), so existing files load, and files written here stay readable by tools that only know those lists. ## Message One conversation turn. Keys, all optional except `id`: | Key | Type | Meaning | |---|---|---| | `id` | string | Message id. | | `user` | list of parts | Content from the user. | | `assistant` | list of parts | Content from the model. | | `toolCalls` | list of tool calls | Client tool calls made in this turn, with results. | | `reasoning` | list of reasoning entries | Provider reasoning kept for replay. | | `dateTime` | RFC 3339 timestamp | When the turn happened. | | `source` | object | Who produced the message (`MessageSource`). | | `cache` | bool | Cache breakpoint after this message. | | `usage` | object | Tokens and cost of the call that produced it. | | `when` | string | Template deciding whether the message is used (initial prompts). | | `partOrder`, `extraParts` | | The order index, below. | In memory, a message's assistant side is **one ordered sequence of parts** (`Message.Assistant`) plus tool calls placed in it by `ToolCall.After` (how many assistant entries precede the call). On disk the sequence is split into the three lists above, and the order index puts it back together. ## Parts A part is one object whose keys say the kind. A part has exactly one kind key. | Kind | Key(s) | Notes | |---|---|---| | Text | `text`, optional `annotations` | Annotations carry citations (`type`, `url`, `title`, offsets, ...). | | Asset | `asset` (object: `storage`, `key`, `hash`, `contentType`, `size`, `name`; `data` only on input, never written) | Content lives in the asset store. | | Reasoning | `reasoning`, `provider`, `display`, `data` | `data` is opaque provider content replayed verbatim (signatures, encrypted content). | | HostedToolCall | `hostedToolCall` (tool name), `provider`, `id`, `status`, `inputs`, `result`, `cost`, `data` | A tool the provider ran. | | Refusal | `refusal` | The model declined. | | Json | `json` | A structured value. | | FileRef | `fileRef`, `provider`, `contentType`, `name`, `uri` | A file already uploaded to a provider. | | ProviderPart | `providerPart` (type), `raw` | Provider-specific content kept as raw JSON. | **Legacy inline media.** A part carrying `image`, `audio`, `video` or `pdf` keys (inline content from older files) is read as an Asset part with no storage reference; the original keys stay in its extension data so they are written back unchanged. Only `asset` is written for new content. ## Tool calls `id`, `tool`, `routine`, `inputs` (object, key order kept), `result` (list of parts), `cost` (number, USD), `when`, `injected`, `injectedBy`. OneLLM adds `isError` (true when the tool failed) and `costLines` (the line items behind `cost`). ## Order index `partOrder` is a list of tokens naming the turn's entries in order: `a` = `assistant[i]`, `r` = `reasoning[i]`, `t` = `toolCalls[i]`, `x` = `extraParts[i]`. `extraParts` holds parts that have no counterpart in the lists (hosted tool calls, refusals, JSON and provider parts, file references on the assistant side). - The index is omitted when the order is the default: all reasoning, then all assistant parts, then all tool calls. - A reader that ignores `partOrder` still sees every list unchanged. The lists are never reordered or padded by the index. - A reader that finds a missing or stale index uses the default order. ## Usage and cost `usage`: `model`, `serviceTier`, `inputTokens`, `outputTokens`, `cacheCreationInputTokens`, `cacheReadInputTokens`, `cost`, and OneLLM's `counters` (map of further counters: `reasoningTokens`, `images`, `audioSeconds`, ...) and `costLines`. **`inputTokens` excludes cached tokens** (reads and writes are separate counters). Provider codecs that receive a total including cache subtract it. Total input = the three summed. A cost line: `counter`, `quantity`, `per`, `unitPrice`, `multiplier`, `amount` (= quantity / per × unitPrice × multiplier), `label`, `source`, `asOf`. **All money is exact decimal.** On disk the cost is a number; nothing is rounded except for display. ## Extension data (unknown fields) Every object above (message, part, tool call, source, usage, cost line, annotation, reasoning entry) keeps keys it does not know and writes them back, at the same level, unchanged in value. Values read from a file are plain: strings, booleans, integers, decimals, lists and mappings. Absence and `null` are not distinguished for unknown keys. ## Timestamps RFC 3339 with a `Z` or numeric offset; fractional seconds are kept to 100 ns. A YAML writer quotes timestamps that would otherwise read as YAML timestamps. ## Stream events A provider's stream is turned into typed events, each naming the content block by `index`: `MessageStart`, `TextDelta`, `ReasoningDelta`, `ReasoningEnd` (provider, opaque data), `RefusalDelta`, `ToolCallStart` (id, tool), `ToolCallArgumentsDelta`, `ToolCallEnd`, `HostedToolCallStart`/`ArgumentsDelta`/`End`, `Annotation`, `PartAdded`, `Usage`, `Done`. Text and argument deltas are UTF-8 slices of the receive buffer, so decoding never allocates a string per event. The **accumulator** builds a message from events, and **streaming and non-streaming calls build equal messages**: - Deltas for one index append in arrival order; a block's position in the turn is the order its first event arrived. - Tool call arguments are concatenated and parsed as JSON when the call ends. A text that is not a JSON object is kept as invalid inputs instead of being dropped. - Usage events merge (a later non-zero counter replaces the earlier value); `Done` ends the message and records the finish reason. - The message id and timestamp are left empty for the runtime to stamp. # Catalog format (version 1) The catalog says which models exist, what they can do and what they cost. It is a set of `llm-provider.yaml` files merged in layers. ## Provider entry One YAML document per provider (a file may hold several, or a `providers:` list): | Key | Meaning | |---|---| | `name` | Provider id, unique across layers. | | `displayName` | For people. | | `modelPatterns` | Globs (`*`, `?`) of model names the provider serves. | | `aliases` | Short name → model name. | | `apiKeyEnvVar` | Environment variable holding the key. | | `capabilities` | What the provider offers (chat, streaming, tools, embeddings, ...). | | `presets` | Named variants of the entry for OpenAI-compatible servers (own base URL, patterns, models). | | `models[]` | `name`, `contextWindow`, `maxOutputTokens`, `supports*` flags, `thinking`, `serviceTiers`, image limits, `pricing`. | | `properties` | Free-form map for anything else. | Keys not listed are kept as extension data and written back. ## Layers Layers are stacked lowest first: **shipped** (embedded in the package) < **user** (files or a directory named by code or an environment variable) < **code** (overrides set programmatically). Providers match by `name`. Merging a provider across layers: - A scalar set in a higher layer replaces the lower value. - Mappings (`aliases`, `properties`, `presets`, `serviceTiers`, `pricing`, ...) merge key by key. - `models` merge **by model name**: a layer can add a model or change single fields of an existing one. - Any other list (`modelPatterns`, `efforts`, price `items`) is replaced as a whole. - A field a layer omits is unchanged. Every effective value records which layer set it (`SourceOf(provider, "models[name].pricing.inputPerMillion")`). ## Resolving a model name 1. An optional `provider:model` or `preset:model` prefix limits the search. A colon belongs to the model name unless the text before it names a provider or preset (`llama3:8b` is one name). 2. Alias. 3. Exact model name. 4. Patterns; the pattern with the most literal characters wins. Names compare ignoring case. A name two providers match at the same step is ambiguous and is an error naming the candidates. An unknown name is a `ModelNotFoundException`. ## Pricing All amounts are USD, exact decimals. - **Per-million token prices:** `inputPerMillion` (tokens that are not cache reads or writes), `outputPerMillion`, `cacheCreationPerMillion`, `cacheReadPerMillion`, and the long-context pair with `longContextThreshold` (applies to the whole request when its total input is above the threshold). - **`items`:** a price for any other usage counter: `counter`, `price`, `per` (default 1,000,000 for counters ending in `Tokens`, else 1), optional `when` qualifiers (`size`, `quality`), `label`, `asOf`, `source`. Counters include `cacheCreation1hInputTokens`, `inputAudioTokens`, `inputImageTokens`, `images`, `audioSeconds`, `characters`, `toolCalls.`, `containerHours`. - **`tiers`:** from a size upwards (`above` of `basis`, default total input tokens) multiply named counters' prices. - **`multipliers`:** named factors (`batch`, `region:eu`) a caller applies. - **`asOf` and `source`:** the date each price was checked and the page it came from, on the model or on a single item. A catalog whose prices are older than 180 days warns. The calculation turns a usage report into cost lines (`counter × price × multipliers`), one line per counter, plus a total. # Records: format v1 Records are an append-only audit trail of a run. Instance state is the truth; records show how it got there. Sinks receive records; a sink that fails never fails a run. ## JSON lines The JSONL sink writes one JSON object per line, to `/.jsonl` (`_runtime.jsonl` for records with no instance). Every line has: | Key | Meaning | |---|---| | `kind` | The record kind, below. | | `time` | RFC 3339 UTC time, from the runtime's clock. | | `instanceId` | The instance. Absent for runtime-level records. | | `messageId` | The message the record belongs to, when there is one. | and then the fields of its kind. Money is a JSON number written from a decimal, never a binary float. | Kind | Fields | |---|---| | `run` | `event` (`started`, `stopped`, `resumed`, `cancelled`, `stateChanged`), `state`, `model` | | `message` | `message`: the message, with its `source` | | `inference` | `provider`, `operation`, `durationMs`, `finishReason`, `usage` (with `costLines`) | | `toolCall` | `call`: the tool call with inputs, result, `cost` and `costLines`; `durationMs` | | `taskCompletion` | `tool`, `result` | | `approval` | `toolCallId`, `tool`, `decision` (`requested`, `approved`, `denied`), `by` | | `compaction` | `strategy`, `at`, `replacedMessageIds`, `summaryMessageId`, `cost` | | `hook` | `point`, `hook`, `decision` (`modify`, `block`, `requireApproval`, `respond`), `reason` | | `error` | `errorType`, `message`, `where` | | `rawTurn` | `request`, `response`: the raw bodies with secrets masked. Off unless the host turns it on. | Readers must ignore kinds and fields they do not know. ## Sinks Several sinks can be active at once. Each record goes to every sink in the order they were added. When a sink throws, the failure is reported to a callback and the record still goes to the other sinks. Cancellation is the one exception that reaches the caller. # Results What OneLLM measured about itself for the 1.0 release, and how. Every number below is OneLLM's own. Run date: 2026-11-06. Commit: the 1.0.0 tag. Machine for the speed rows: AMD Ryzen 9 7950X3D, 16 physical and 32 logical cores, 125 GB, Linux Mint 22.3 (kernel 7.0.0-38), .NET SDK 10.0.403, runtime 10.0.14, BenchmarkDotNet 0.15.8, on an otherwise idle machine (1-minute load average between 1.1 and 8.4 during the runs). ## Speed Method: BenchmarkDotNet suites in `bench/` against a local mock server that replays deterministic provider responses over loopback, so the numbers are client code and not a network or a vendor. The workloads, the exact settings and the way the gate runner records the machine and its load are in [`bench/README.md`](https://github.com/recursive-ai-oss/onellm-csharp/blob/38ccb22024da28a34dac5d87d1ff83fae2656dee/bench/README.md). Each row is the mean of the measured iterations; allocations are bytes per operation from the memory diagnoser. A request row is a whole non-streamed turn: build the request from a history of the stated size, serialize it, send it over loopback and parse a one-line answer. A stream row reads a 2,000-event text stream (divide by 2,000 for the per-event cost); a first-event row stops at the first text event of a 50-event stream. | Row | Parameters | Mean | Allocated | Peak working set (MB) | |---|---|---:|---:|---:| | request-anthropic | 1k tokens | 50.0 us | 31.7 KB | 146 | | request-anthropic | 50k tokens | 1.022 ms | 714.1 KB | 91 | | request-anthropic | 200k tokens | 5.788 ms | 2.75 MB | 106 | | request-gemini | 1k tokens | 37.0 us | 27.1 KB | 141 | | request-gemini | 50k tokens | 779.1 us | 839.0 KB | 88 | | request-gemini | 200k tokens | 1.835 ms | 3.27 MB | 100 | | request-openai-chat | 1k tokens | 35.9 us | 15.1 KB | 164 | | request-openai-chat | 50k tokens | 652.9 us | 291.2 KB | 89 | | request-openai-chat | 200k tokens | 1.301 ms | 1.11 MB | 91 | | request-openai-responses | 1k tokens | 38.1 us | 21.5 KB | 163 | | request-openai-responses | 50k tokens | 692.5 us | 297.7 KB | 86 | | request-openai-responses | 200k tokens | 1.422 ms | 1.12 MB | 91 | | stream-anthropic | 2,000 events | 610.7 us | 33.7 KB | 105 | | stream-gemini | 2,000 events | 699.5 us | 155.5 KB | 131 | | stream-openai-chat | 2,000 events | 813.7 us | 639.7 KB | 140 | | stream-openai-responses | 2,000 events | 621.0 us | 96.3 KB | 116 | | first-event-anthropic | first text event | 54.8 us | 31.9 KB | 144 | | first-event-gemini | first text event | 43.4 us | 29.4 KB | 143 | | first-event-openai-chat | first text event | 45.7 us | 13.8 KB | 163 | | first-event-openai-responses | first text event | 52.6 us | 22.2 KB | 147 | ### The agent loop One run is eleven model turns (ten tool-calling turns and a final one) on the scripted provider, with the default runtime, an in-memory store and no record sinks: **78.2 us and 56.8 KB per run, 7.1 us and 5.2 KB per turn**. That is above the fastest managed reference by a factor of **7.92**, because every OneLLM run is a persisted instance with a snapshot, priced line items and events for each turn. Floors, set at this first measurement and not to rise in later releases: | Floor | Value | |---|---| | Per-turn save of a 200,000-token instance (append one turn, atomic file replace) | 7.22 ms, 4.39 MB allocated | | Run with no hook (the eleven-turn run above) | 79.2 us, 56.8 KB | One observer hook on the message point adds about 0 us and 1.8 KB per run (78.6 us, 58.6 KB). Keeping the retained instance in memory costs 79.3 us per run. The same run with the file store, which writes the instance after every turn, takes 94.5 ms and 621 KB. ### Other costs | Row | Mean | Allocated | |---|---:|---:| | Serialize a 200,000-token instance with one more turn | 544 us | 3.52 MB | | Load a 200,000-token instance | 1.98 ms | 7.61 MB | | Save a never-written 200,000-token instance | 1.86 ms | 5.54 MB | | Template: plain text / field substitution / range and if / embed | 47 ns / 210 ns / 4.6 us / 309 ns | | | Template: cached parse lookup / cold parse | 55 ns / 7.6 us | | | Build the shipped catalog / build and resolve one model | 1.47 ms / 1.49 ms | 2.26 MB | | Token count, core estimate, 1k / 50k / 200k tokens | 2.2 us / 58 us / 388 us | | Core ships no local tokenizer; its count comes from the provider's endpoint when one exists and otherwise from the calibrated estimate timed above. ## Conformance Each feature of each provider, on each deployment that has one, is replayed from recorded fixtures: the request bytes OneLLM sends and the result it parses are both checked. Streaming and whole-response calls must build equal messages. The core test suite (1,304 tests: 1,239 passed, 65 skipped for platform reasons, 0 failed) includes all of these. | Provider | Deployment | Fixture files replayed | Failed | |---|---|---:|---:| | Anthropic | first-party | 47 | 0 | | Anthropic | Amazon Bedrock | 9 | 0 | | Anthropic | Vertex AI | 6 | 0 | | OpenAI | first-party | 67 | 0 | | OpenAI | Azure OpenAI | 7 | 0 | | Gemini | Gemini API | 57 | 0 | | Gemini | Vertex AI | 6 | 0 | | OpenAI-compatible | nine presets, two files each | 18 | 0 | TypeSafe Jev is covered by the decider checks below. The Bedrock request signing passes the published SigV4 test suite, and the Vertex service-account token passes the published JWT vector. ## Parity and round trips The reference corpus of definitions and instances is a private collection kept on the build machine; only counts are reported. | Check | Result | |---|---| | Definition, flow, decider and instance files load and save without loss (extension data kept) | 5,473 files: 5,440 lossless, 0 lossy, 24 invalid input (a raw U+FEFF inside a block scalar), 9 removed during the run | | Rendered prompts match the recorded rendering byte for byte | 600 sampled: 328 byte-equal, 271 not reproducible (scope not recorded, or sources changed since), 1 differing because the definition was edited after it was rendered; 0 engine differences | | Template engine agrees with Go's `text/template` on generated templates (output and error class) | 12,858,638 templates in 1,680 s: 124 disagreements, of which 6 were fixed and 118 are documented differences (slices beyond the length but within capacity; strings cut mid-character) | | Decider requests are byte-equal to the reference service's, and `decide` outputs agree | 336 of 336 requests byte-equal; outputs 144 of 144 for JavaScript logic and 144 of 144 for TypeScript logic; 35 of 35 TypeScript logic files load | | Cost agrees with the reference calculator to the cent | 1,260 of 1,260 usages | ## Loop behaviour, security and AOT | Check | Result | |---|---| | Loop scenarios (termination, parallel calls in call order, approvals across a restart, steering, sub-agents, monitors, compaction pairing, cache-prefix stability), on the scripted provider and a fake clock | 93 agent tests, all passed | | Capability-model review | 28 findings (CAP-1 to CAP-28), all closed; documented residue: detached children after a timeout with an open shell, hard links, an allow-list entry is not a sandbox, the cost of the git role rule | | Sandbox review | 18 findings: 17 closed; one low (a burst of parallel hook-file loads across many definitions) bounded per owner and per definition, documented | | NativeAOT publish of the sample apps (core: `Ask`, `Ask`, an agent with generated tools, a file store; JavaScript; OneLite; the Microsoft.Extensions.AI bridge) | 0 warnings; all four run; the same samples also build and run in CI on Windows x64, Linux x64, Linux arm64 and macOS arm64 | | Fuzz targets | Event-stream reader: 600 s, 47.4 million inputs, 0 findings. Template parser: 3.19 million inputs, 0 findings; the template differential above adds 12.86 million generated templates | ## Live runs A small smoke against each provider's real API, run once at release under hard spend caps (`tools/live-smoke.sh`): chat, streaming, a client tool round trip, structured output, a reasoning model, embeddings, token counting and, for OpenAI, a Decisions call. Then each of the eight example agents ran its representative task once on an OpenAI and a Gemini model (`tools/examples-e2e.sh`). The first OpenAI smoke passed 7 of 8 (the Decisions case used a wrong model name, fixed in the tool); the second passed 8 of 8. | Provider | Smoke | Spent in total (cap) | |---|---|---| | OpenAI | 8 of 8 cases passed | $0.0264 ($0.90) | | Gemini | 7 of 7 cases passed | $0.198 ($0.45) | | Anthropic | **Not run**: no API key was available for the release check | n/a | | Azure OpenAI / Foundry | **Not run**: no Azure account; covered by recorded fixtures only | n/a | | Amazon Bedrock | **Not run**: no AWS account; covered by recorded fixtures and the published SigV4 test vectors | n/a | | Vertex AI | **Not run**: no Google Cloud account; covered by recorded fixtures and the published JWT vector | n/a | Example agents, one run each. Outcome `task_complete` means the agent ended by calling `task_complete`; `answered` means it replied without calling it. | Example agent | Model | Outcome | Turns | Cost | Seconds | |---|---|---|---:|---:|---:| | `cs-tool-writer` | gpt-5-nano | answered | 2 | $0.003912 | 77 | | `js-tool-writer` | gpt-5-nano | task_complete | 8 | $0.003624 | 72 | | `provider-writer` | gpt-5-nano | task_complete | 5 | $0.001828 | 42 | | `agent-writer` | gpt-5-nano | answered | 1 | $0.001523 | 35 | | `script-writer` | gpt-5-nano | task_complete | 1 | $0.001398 | 32 | | `decider-writer` | gpt-5-nano | answered | 4 | $0.003123 | 64 | | `hook-writer` | gpt-5-nano | answered | 1 | $0.002322 | 51 | | `orchestrator` | gpt-5-nano | task_complete | 13 | $0.007200 | 130 | | `cs-tool-writer` | gemini-3.5-flash-lite | task_complete | 26 | $0.030503 | 27 | | `js-tool-writer` | gemini-3.5-flash-lite | task_complete | 13 | $0.016682 | 11 | | `provider-writer` | gemini-3.5-flash-lite | task_complete | 19 | $0.022865 | 15 | | `agent-writer` | gemini-3.5-flash-lite | task_complete | 9 | $0.011340 | 8 | | `script-writer` | gemini-3.5-flash-lite | task_complete | 8 | $0.009396 | 6 | | `decider-writer` | gemini-3.5-flash-lite | task_complete | 16 | $0.022134 | 17 | | `hook-writer` | gemini-3.5-flash-lite | task_complete | 25 | $0.033729 | 31 | | `orchestrator` | gemini-3.5-flash-lite | task_complete | 13 | $0.016196 | 11 | The Gemini `cs-tool-writer` row is its second attempt; the first stopped at its per-run spend cap after $0.0348, which is counted in the Gemini total above. A row is a full success when the agent calls `task_complete`, the files its task names exist, and its own checks passed (see [the example agents](https://recursiveai.org/onellm/docs/examples/)); the `answered` rows are runs of the small OpenAI model that stopped without finishing the protocol. # Changelog All notable changes to this project are documented in this file. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project follows [Semantic Versioning](https://semver.org/). ## [1.0.0] - 2026-11-06 The first public release. ### Added - **`RecursiveAI.OneLlm`** (core; depends on the .NET base library, OneYaml and OneImage): - Provider clients for Anthropic Messages, OpenAI (Responses; Chat Completions for compatible servers) and Gemini `generateContent`: chat, streaming, client and hosted tools, structured output, reasoning replay, caching, token counting, batch and embeddings. OpenAI Decisions and TypeSafe Jev for deciders. OpenAI-compatible presets for Ollama, vLLM, LM Studio, OpenRouter, Groq, DeepSeek, xAI, Mistral and Together. - Deployments: Azure OpenAI / Foundry, Amazon Bedrock (SigV4 and the event stream) and Vertex AI (service-account JWT) for Claude and Gemini. See `docs/deployments.md`. - Image generation, text-to-speech, transcription and moderation clients, each with a costed built-in tool. - One turn-shaped message model with a closed part union. Stream events build the same message a whole call returns, and unknown fields are kept as extension data. - A layered model catalog in `llm-provider.yaml` form (shipped files, then user files, then code), with `decimal` line-item pricing, `asOf` dates and source URLs. - Three-tier token counting that reports its tier. - Typed errors, retries that honour `Retry-After`, and `Secret` scrubbing. - Prompt templates in the Go `text/template` language, with a documented function set, cached parse trees and positioned errors. Known differences from Go are listed in `docs/templating.md`. - `Runtime` and the `Llm` happy path: ask, typed ask, streaming, embeddings, chat, scripted runs, agents and deciders. Every run is an instance with cost, records and telemetry. - A scripted provider for tests (`OneLlm.Testing`). - Agents: - the tool loop, with parallel calls returned in call order; - persisted approvals, steering and sub-agents; - monitors and context strategies; - C# hooks. - Tools: the `ITool` contract and the `[Tool]` source generator. The signature-to-JSON-Schema table is in `docs/tool-schemas.md`. - Built-in tools for files, shell, web, agents, judgment, media and working state. They run under a host-granted capability model (file roots, shell, network egress). A definition never grants a capability. - Deciders: - `decider.yaml` definitions, with three engines chosen through the catalog; - `forEach` and budget splitting, typed refusals, and input and output schema checks; - runs with replay, list and statistics; - C# deciders with generated typed answers (`[Decider]`), and `Llm.DecideAsync`. - Instance stores (file, memory) and record sinks (JSONL, events). OpenTelemetry GenAI spans and metrics. - **`RecursiveAI.OneLlm.JavaScript`** (on OneEcma 1.1.1): - Covers JavaScript tool files, the `run_javascript` built-in, JavaScript hooks, and decider logic in `decider.js` or `decider.ts` (types erased in place, errors pointing into the `.ts` file). - Everything runs in a sandbox with fuel, allocation, wall-clock and string limits, per-run quotas, and a shared limiter that keeps capacity for hooks and decider logic. See `docs/javascript.md`. - **`RecursiveAI.OneLlm.Web`:** `fetch_webpage_markdown`, on OneHtml. - **`RecursiveAI.OneLlm.Extensions.AI`:** the Microsoft.Extensions.AI bridge, in both directions. - OneLLM models as `IChatClient`, `IEmbeddingGenerator`, `ISpeechToTextClient`, `ITextToSpeechClient` and `IImageGenerator`, with usage and cost in `AdditionalProperties`. - Any `IChatClient` or embedding generator as a OneLLM provider. - `AIFunction` and `ITool` in both directions. - **`RecursiveAI.OneLlm.EntityFrameworkCore`** and **`RecursiveAI.OneLlm.OneLite`:** an instance store and a record sink each. - **`RecursiveAI.OneLlm.Cli`:** the `onellm` tool, with `run`, `chat`, `instances`, `cost`, `catalog` and `examples`. Grants are given as explicit flags. It embeds eight example agents. - **Documentation:** getting started; per-feature guides; the "reads like" tables (`docs/reads-like.md`); per-package readmes; and a results page (`docs/results.md`) with the speed, conformance, parity, AOT and live-run results and their method. ### Security - The capability model and the JavaScript sandbox were each reviewed before this release, and every finding is closed or documented in `docs/capabilities.md` and `docs/javascript.md`. Those pages also list what the model does not protect against, for example hard links, and an allow-listed program that can itself run arbitrary commands. - With `git` on the shell allow-list, either directly, behind a launcher, or through a git front end, git configuration and hook files are refused to the file tools.