diff --git a/AGENTS.md b/AGENTS.md index 622abd801b48..ae585a898a94 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,19 +1,16 @@ - After changing the public Protocol or Server `HttpApi`, run `bun run generate` from `packages/client`. Do not edit generated client files directly. - Keep runtime dependencies directed from Schema to Core and Protocol, then from Core and Protocol to Server. Client runtime code may depend on Schema and Protocol but never Core or Server; `sdk` composes Client, Core, and Server. -- Current implementation changes belong in `packages/core`, `packages/cli`, `packages/server`, `packages/protocol`, `packages/schema`, and related generated client surfaces when required. - This repository does not use Changesets. Do not add `.changeset` files; follow the existing release workflow instead. -- The default branch in this repo is `dev`. -- Default new branches and worktrees to `dev`, or `origin/dev` when the local `dev` ref is unavailable, and default pull requests to target `dev`. Use another base or target branch when the requester explicitly instructs it. -- Local `main` ref may not exist; use `dev` or `origin/dev` for diffs. +- The default branch is `dev`. Base branches and worktrees on `origin/dev`, target PRs at `dev`, and diff against `origin/dev` unless told otherwise. -## Live V2 TUI Testing +## Live TUI Testing - Run `bun run dev:live` from a development worktree to test its TUI against the currently elected `opencode` background server and live sessions. - Pass a directory after the script when needed, for example `bun run dev:live /path/to/project`. - The script discovers the server with `opencode service status`, injects its private local credential from `opencode service get password`, and uses the `dev` TUI storage channel so tabs and other client-local state match the installed client. - Prefer `dev:live` over plain `bun run dev` for this workflow. An implicit managed-service connection may replace the live server when the worktree client version differs; explicit `--server` warns and continues without replacing it. -## V2 TUI Stories +## TUI Stories - When a user asks for a TUI story, add a fixture-driven story under `packages/tui/src/feature-plugins/system/storybook` and register it in `index.tsx`. - Render the real production component rather than a visual copy. Keep submissions and other side effects local to the story so it is safe to explore repeatedly. @@ -54,7 +51,7 @@ Examples: `fix(tui): simplify thinking toggle styling`, `docs: update contributi - Use Bun APIs when possible, like `Bun.file()` - Rely on type inference when possible; avoid explicit type annotations or interfaces unless necessary for exports or clarity - Prefer functional array methods (flatMap, filter, map) over for loops; use type guards on filter to maintain type inference downstream -- In `src/config`, follow the existing self-export pattern at the top of the file (for example `export * as ConfigAgent from "./agent"`) when adding a new config module. +- In `src/config`, follow the existing self-export pattern at the top of the file (for example `export * as ConfigAgent from "./agent.js"`) when adding a new config module. - In Effect generators, bind services to named variables before calling methods. Do not use nested service yields such as `yield* (yield* Foo.Service).bar()`. Reduce total variable count by inlining when a value is only used once. @@ -175,18 +172,15 @@ const table = sqliteTable("session", { - Run `bun run check` from the repository root as the canonical full lint and type-check verification. - During focused iteration, run `bun typecheck` from the affected package directory (for example, `packages/core`). Never run `tsc` directly. -## V2 Session Core +## Session Core - Keep durable events minimal: record irreducible new facts and do not repeat state derivable by folding the ordered aggregate history. Enrich projections and read models with previous or derived state when consumers need self-contained views. - Keep durable prompt admission separate from model execution. `Session.prompt(...)` publishes `session.inbox.enqueued`, whose projection inserts one durable `session_inbox` row, before scheduling advisory `SessionExecution.wake(sessionID)` unless `resume: false` requests admit-only behavior. Delivery publishes `session.inbox.delivered`; its projection consumes the inbox row and inserts the visible message in the same transaction. `session_inbox` stores only unconsumed work. - Reusing a Session ID adopts the existing Session. Reusing a user or synthetic inbox item ID is idempotent when Session and type match: the first admission wins and the retried payload, metadata, and delivery mode are ignored, whether the item is still pending or already delivered (reconciled from the projected message without retained enqueue history). Cross-Session or cross-type reuse fails. Control items keep their operation-specific conflict behavior. -- Keep `SessionExecution` process-global and Session-ID based. Its local implementation owns the process-local Session coordinator and discovers placement through `SessionStore` plus `LocationServiceMap.get(session.location)` only when a drain starts; no layer should take a Session ID. V2 interruption targets the active process-local ownership chain for that Session; interruption of a known but idle or locally unowned Session is a no-op, while the public API rejects an unknown Session. +- Keep `SessionExecution` process-global and Session-ID based. Its local implementation owns the process-local Session coordinator and discovers placement through `SessionStore` plus `LocationServiceMap.get(session.location)` only when a drain starts; no layer should take a Session ID. Interruption targets the active process-local ownership chain for that Session; interruption of a known but idle or locally unowned Session is a no-op, while the public API rejects an unknown Session. - Keep `SessionRunner`, model resolution, tool registry, permissions, and filesystem Location-scoped. Omitted `Location.workspaceID` means implicit-local placement; explicit workspace identity remains reserved for future placement semantics. -- Preserve one explicit `llm.stream(request)` call per Physical Attempt and reload projected history before durable continuation. A logical Step may use generic pre-output retries, one full-context retry after continuation rejection, incomplete-stream continuation, or one overflow-compaction rebuild. Generic retries retain the logical step number and do not consume another agent-step allowance. Do not delegate orchestration to an in-memory tool loop. +- Each step is one explicit `llm.stream` call that continues from reloaded history. Do not add an in-memory tool loop. - Keep local Session drains process-local until clustering is implemented. `SessionRunCoordinator` joins explicit same-Session resumes, coalesces prompt wakeups, and allows different Sessions to run concurrently. A write-ahead execution claim marks a process-local busy period for restart recovery: terminal completion, failure, or user interruption releases it, while shutdown interruption and process death preserve it. Startup recovery resumes claimed top-level Sessions with durable per-execution attempt accounting. The claim is a recovery marker, not clustered ownership, fencing, or an exactly-once guarantee. -- Keep provider-specific native compaction mechanisms in `@opencode/ai` behind `LLMClient.compact`. `SessionCompaction` chooses a summary or native compaction from the model's `compaction` setting and owns route provenance, request shrinking, the retry policy, interruption, usage accounting, and checkpoint persistence. - Keep delivery vocabulary explicit. Prompts steer by default. At safe step boundaries, steered compaction takes priority up to the first steered move control; other steers retain enqueue order. At an idle boundary, steers take priority; otherwise exactly one queued item delivers before the runner reevaluates continuation. Inbox items may be cancelled or changed between queue and steer before delivery. Promoting new user input resets the selected agent's step allowance; a batch of steers resets it once. - One step is one logical LLM call; its durable record covers only the model-visible span. Do not write "provider turn", and do not use bare "turn" for a single call: "turn" is reserved for the future assistant-turn unit containing all steps from prompt promotion until the session would go idle. - Keep event replay ownership separate from clustered Session execution ownership. -- Keep the Instructions algebra and built-ins in `src/instructions`; keep instruction producers with their observed domains, and keep Session History selection plus `InstructionState` and `InstructionEntry` persistence Session-owned. `InstructionDiscovery` observes ambient global and upward-project instructions. The runner composes built-ins, discovery, guidance, and entries explicitly in `loadInstructions`; there is no instruction registry. -- `session.instructions.updated` stores changed source keys and content hashes and may freeze rendered chronological update text. Blob values live once in `instruction_blob`; the projected `instruction_state` row is the normal boundary-processing source of current and initial values. Request assembly renders the epoch baseline from stored values, while later frozen updates enter history as durable System messages. Completed compaction moves the instruction epoch; Session movement retains it so destination instruction changes are chronological, while committed revert clears it. Forks adopt the parent's newest instruction values even when copied message history ends at an earlier boundary. Unavailable sources retain the last value and block only the initial complete delta. diff --git a/packages/ai/AGENTS.md b/packages/ai/AGENTS.md index 3547ca092116..d5a33c4875e0 100644 --- a/packages/ai/AGENTS.md +++ b/packages/ai/AGENTS.md @@ -1,324 +1,20 @@ # AI Package Guide -## Effect +## Principles -- Prefer `HttpClient.HttpClient` / `HttpClientResponse.HttpClientResponse` over web `fetch` / `Response` at package boundaries. -- Use `Stream.Stream` for streaming data flow. Avoid ad hoc async generators or manual web reader loops unless an Effect `Stream` API cannot model the behavior. -- Use Effect Schema codecs for JSON encode/decode (`Schema.fromJsonString(...)`) instead of direct `JSON.parse` / `JSON.stringify` in implementation code. -- In `Effect.gen`, yield yieldable errors directly (`return yield* new MyError(...)`) instead of `Effect.fail(new MyError(...))`. -- Use `Effect.void` instead of `Effect.succeed(undefined)` when the successful value is intentionally void. +- **Forward compatibility.** Providers ship new models, options, and values constantly. Accept unknown pass-through values (`Known | (string & {})` for string options); use closed literals only when OpenCode branches on the value. +- **One canonical model.** The Schema classes in `src/schema/` are the data model. Do not add a parallel shape or a second construction path. +- **Independent of Session.** Auth, permissions, plugins, and orchestration belong in Core. `LLM.stream`/`LLM.generate` run exactly one model call. +- **Reuse protocols.** Providers compose existing protocols with `Route.make(...)` instead of cloning them. -## Conventions +## Code style -Per-type constructors live on the type, not as top-level re-exports. Use `Message.system(...)`, `Message.user(...)`, `Message.assistant(...)`, `Message.tool(...)`, `Message.media(...)`, `LanguageModel.make(...)`, `ToolDefinition.make(...)`, `ToolCallPart.make(...)`, `ToolResultPart.make(...)`, `ToolChoice.make(...)`, `ToolChoice.named(...)`, `SystemPart.make(...)`, and `GenerationOptions.make(...)` directly. The top-level `LLM` namespace is reserved for request-shaped call APIs: `LLM.request`, `LLM.generate`, `LLM.stream`, and `LLM.generateObject`. `LLM.generate`/`LLM.stream` and Promise `ai.llm.generate`/`ai.llm.stream` accept ergonomic input or an `LLMRequest`; both paths use the same canonical request. Core still builds, logs, replays, and updates that durable `LLMRequest` boundary. Use `LLMRequest.update(...)` when deriving canonical request data; do not add a duplicate `LLM.updateRequest(...)` path. - -Modality namespaces mirror `LLM` exactly: `Image.request`, `Image.generate`, `Image.stream`, and the same for `Video`, `Speech`, and `Transcription`. Common request fields (`images`, `mask`, `n`, `size`, `aspectRatio`, `seed`, `format`) lower natively or fail with a typed `AIError`; provider-native controls always live under `providerOptions`, never under a modality-specific `options` key. - -Media payloads are always `Media.Asset` (`src/media.ts`). Construct them with `Media.bytes`, `Media.base64`, `Media.url`, `Media.ref`, `Media.fromDataUrl`, or `Media.file`; never introduce a parallel `data: string | Uint8Array` shape. `MediaPart.media`, `ImageRequest.images`/`mask`, `ImageResponse.images`, and the `media` `LLMEvent` all share it. Protocols branch on `asset.source.type` and `asset.kind` and use `ProviderShared.requireInlineMedia` / `inlineRequired` / `mediaUrl` / `mediaReference` and `MediaInput.inlineBytes` / `refID` rather than re-deriving base64 or URL handling. - -`schema/messages.ts → media.ts → route/executor-service.ts` is an accepted runtime dependency from the schema layer on the executor service tag: `Media.Asset.bytes()` must be able to download `url` sources, and the tag lives in that leaf module precisely so the schema barrel never imports the executor implementation (which imports the schema barrel back). Do not move the tag into `route/executor.ts` or import `route/executor.ts` from `src/schema/*` or `src/media.ts`. - -Nothing in `src/*` except `src/promise.ts` may know about Promises. `@opencode/ai/promise` (`AI.make({ layer? })`, default `ai`) is the single Promise/`AsyncIterable` surface for LLM and media; it runs the Effect APIs in one `ManagedRuntime` and rethrows `AIError` unchanged. - -- Prefer forward compatibility for provider-defined options that OpenCode only passes through. For pass-through string enums, expose known values for autocomplete while accepting future values with `Known | (string & {})`, and accept any string at runtime. Closed literals are appropriate when OpenCode branches on a value, transforms its associated structure, or otherwise cannot correctly handle an unknown variant. New options whose shape or behavior requires implementation remain unsupported until they are handled; do not blindly forward unknown structures. -- Order reasoning-effort values from lowest to highest: `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`. Provider-specific subsets follow the same relative order in types, schemas, option lists, and tests. +- Use `HttpClient`, `Stream`, and Schema codecs instead of `fetch`, async generators, and `JSON.parse`/`JSON.stringify`. +- `return yield* new MyError(...)` instead of `Effect.fail(...)`; `Effect.void` instead of `Effect.succeed(undefined)`. +- Only `src/promise.ts` may use Promises. ## Tests -- Use `testEffect(...)` from `test/lib/effect.ts` for tests requiring Effect layers. -- Keep provider tests fixture-first. Live provider calls must stay behind `RECORD=true` and required API-key checks. - -## Errors - -- `AIError` wraps a union of tagged reason errors. It stores only `reason`, derives `message` from the reason, and exposes the reason as its `cause`. -- Each reason owns its readable `message`, category-specific fields, and optional `body`, `http`, and underlying exception in `cause`. -- `reason.body` is the sole original-response or triggering-event payload field. Preserve original text before schema decoding removes fields; do not replace the complete event with only its nested error. -- `reason.http` describes an observed HTTP response with required `url`, `status`, and response `headers`. Do not invent status codes or derive a separate request ID from headers. -- Reclassification and transport recovery must preserve the reason's body, HTTP context, and underlying cause. Error `message` and `cause` are non-enumerable: copy them explicitly when constructing an enriched reason with its constructor or `AIErrorReason.make`. - -## Architecture - -This package is an Effect Schema-first LLM core. The Schema classes in `src/schema/` are the canonical runtime data model. Convenience functions in `src/llm.ts` are thin constructors that return those same Schema class instances; they should improve callsites without creating a second model. - -Session integration lives in `packages/core/src/session`: `runner/llm.ts` owns orchestration, `model-request.ts` lowers Session state into `LLMRequest`, and `model-transport.ts` selects transport behavior. - -Keep this package independent of Session concerns. Session auth, permissions, plugins, telemetry headers, and runtime selection belong in Core. - -### Request Flow - -The intended callsite is: - -```ts -const request = LLM.request({ - model: OpenAI.configure({ apiKey }).responses("gpt-4o-mini"), - system: "You are concise.", - prompt: "Say hello.", -}) - -const response = yield * LLMClient.generate(request) // inside Effect.gen -``` - -`LLM.request(...)` builds an `LLMRequest`. `LLMClient.generate(...)` reads the executable route carried by `request.model.route`, builds the provider-native body, asks the route's transport for a real `HttpClientRequest.HttpClientRequest`, sends it through `RequestExecutor.Service`, parses the provider stream into common `LLMEvent`s, and finally returns an `LLMResponse`. - -Use `LLMClient.stream(request)` when callers want incremental `LLMEvent`s. Use `LLMClient.generate(request)` when callers want those same events collected into an `LLMResponse`. - -Filter or narrow `LLMEvent` streams with `LLMEvent.is.*` (camelCase guards, e.g. `events.filter(LLMEvent.is.toolCall)`). The kebab-case `LLMEvent.guards["tool-call"]` form also works but prefer `is.*` in new code. - -### Routes - -A route is the runnable composition of four orthogonal pieces: - -- **`Protocol`** (`src/route/protocol.ts`) — semantic API contract. Owns request body construction (`body.from`), the body schema (`body.schema`), the streaming-event schema (`stream.event`), and the event-to-`LLMEvent` state machine (`stream.step`). `Route.make(...)` validates and JSON-encodes the body from `body.schema` and decodes frames with `stream.event`. Examples: `OpenAIChat.protocol`, `OpenResponses.protocol`, `OpenAIResponses.protocol`, `AnthropicMessages.protocol`, `Gemini.protocol`, `BedrockConverse.protocol`. -- **`Endpoint`** (`src/route/endpoint.ts`) — URL construction. The host, path, and route query live on the endpoint. `Endpoint.path("/chat/completions", { baseURL })` is the common case; pass a function for paths that embed the model id or a body field (e.g. `Endpoint.path(({ body }) => `/model/${body.modelId}/converse-stream`)`). -- **`Auth`** (`src/route/auth.ts`) — per-request transport authentication. Provider facades configure credentials onto the route before model selection, usually via `Auth.bearer(apiKey)` or `Auth.header(name, apiKey)`. Routes that need per-request signing (Bedrock SigV4, future Vertex IAM, Azure AAD) implement `Auth` as a function that signs the body and merges signed headers into the result. -- **`Framing`** (`src/route/framing.ts`) — bytes → frames. SSE (`Framing.sse`) is shared; Bedrock keeps its AWS event-stream framing as a typed `Framing` value alongside its protocol. - -Compose them via `Route.make(...)`: - -```ts -export const route = Route.make({ - id: "openai-chat", - provider: "openai", - protocol: OpenAIChat.protocol, - endpoint: Endpoint.path("/chat/completions", { - baseURL: "https://api.openai.com/v1", - }), - auth: Auth.bearer(Auth.config("OPENAI_API_KEY")), - framing: Framing.sse, -}) -``` - -Route defaults are request-shaping defaults such as `headers`, `limits`, `generation`, `providerOptions`, and `http`. Endpoint host/query belongs on the route endpoint. Selected `LanguageModel` values carry only model id, provider id, and the configured route value. Model capability/catalog metadata lives outside this package; protocol support is enforced by request lowering and typed `AIError`s. - -The four-axis decomposition is the reason DeepSeek, TogetherAI, Cerebras, Baseten, Fireworks, and DeepInfra all reuse `OpenAIChat.protocol` verbatim — each provider owns a small `Route.make(...)` composition instead of a protocol clone. Bug fixes in one protocol propagate to every consumer of that protocol in a single commit. - -When a provider supports multiple physical transports, selection remains execution policy below its semantic route. `OpenResponsesChannel.transport(...)` owns the provider-neutral Responses WebSocket concept: it prepares one final request, executes HTTP by default, strips WebSocket-disallowed fields, and passes a generic channel exchange to a per-call `WebSocketChannelExecutor` when supplied. Provider-specific Responses routes opt in with handshake and connection-age policy. `Route.streamPrepared` owns decoding and acknowledges channel completion only after successful full consumption. - -### Media Routes - -Media does not fit the SSE-frames-to-event-state-machine LLM route. `MediaRoute.inline(...)` / `queued(...)` / `stream(...)` (`src/route/media.ts`) compose a `MediaProtocol` kind with `Endpoint` and `Auth` and own the transport plumbing: `http` option merging, URL/query rendering, auth headers, JSON vs multipart encoding, and handing the response back to the protocol. `MediaProtocol.inline` (`src/route/media-protocol.ts`) is `body.from(request)` plus `response.decode(response, context)`; each protocol declares `const route = MediaProtocol.identity({ id, name, provider })` once and decodes through `route.decodeJson` / `route.text` / `route.decodeStarted` so decode failures retain the raw body and HTTP context, raising `route.unsupported(operation, message)` for requests it cannot lower, and passes `route` as the first argument to `MediaProtocol.inline` / `queued` / `stream`. `Generation` (`src/generation.ts`) is the provider-neutral handle for a queued generation over a `GenerationRoute` (`status`, `result`, `cancel`). Image protocol files follow the same section order as LLM protocols and declare unsupported common fields once through the protocol's `unsupported` list. - -`MediaProtocol.queued` is the submit-then-poll kind every video route uses: `start` (body + decode into `{ token, snapshot }`), `status`, `result`, and optional `cancel` (with `activeOnly` when the provider's cancel endpoint deletes finished work, as Runway's does: the route refreshes status first and skips terminal generations), each addressed by a route-owned `token` whose `Schema.Codec` makes it serializable. `MediaRoute.inline` and `MediaRoute.queued` compose the two kinds with `Endpoint` and `Auth`; the queued route decodes the token once at the boundary (`start` output or `resume` input) and closes over it in a token-free `GenerationRoute` (`status`/`result`/`cancel` are plain Effects), so `Generation` never sees the token's shape and only carries the encoded JSON for persistence. Polls reuse the route's auth and deployment headers plus the request's `http` overlay after `start`, and resolve relative paths against the route base URL (provider-issued absolute URLs such as fal's `status_url` pass through). `result` is always its own GET even when the provider returns output inside the status document, so `Generation.await` behaves the same after `start` and after `resume`. `PollContext.auth` carries only what `Auth` added or changed so protocols can hand download credentials to output assets as transient `Media.Asset.headers` (Veo) — never part of `source` or JSON. Status strings map through a per-protocol `STATUS` table via `MediaProtocol.status`; terminal generations without output fail through `output.ended` / `output.contentPolicy` with the provider document on `reason.body`; a `failed` generation maps the provider's error code through a per-protocol `FAILURE` table via `MediaProtocol.failure` so rejected inputs are not reported as retryable `ProviderInternal`. `GenerationAwaitOptions` (`AwaitOptions` in `src/generation.ts`, `{ poll?: Poll }`) is the one options type for `await`, `events`, `Video.generate`, and `Video.stream`. - -`MediaProtocol.stream` is the incremental kind every speech route uses, with the same discipline as LLM protocols. `MediaRoute.stream` submits the caller's request as `MediaProtocol.Addressed` (`{ ...request, mode }`, `mode: "generate" | "stream"`), so one provider stays one protocol: `body.from`, the endpoint path, and `frames` read `request.mode` to pick the body, path, and framing. `frames(bytes, context)` returns frames — `Framing.sse`, `Framing.lines`, `Framing.document` (a single-document response shaped like a streamed record), or the raw `bytes` for chunked audio. `initial()` is fresh per-response parser state; `step` folds each frame into it and emits modality events; `finish(state, context)` runs once after the last frame with the request, body, and observed `http` (header-only usage lives there) and emits exactly one terminal event or fails with `route.incomplete()`. Keep parser state to real accumulators and derive anything the request or body determines in `finish`. `generate` runs the same stream and folds it with the modality's `collect`. Request-derived URL parameters go on the body's `query` (array values repeat the parameter), applied before route and caller `http.query`. Decode frames with `route.decodeFrame` and raise stream-time failures with `route.frameError` (the frame stays on `reason.body`); protocols never thread HTTP context, because the route fills `reason.http` on stream errors that lack it. Speech protocols share `protocols/utils/speech-stream.ts` for deltas, timestamps, voice ids, PCM and container descriptions, and the terminal asset. - -Every modality route is the inline | stream | queued union (transcription uses all three: OpenAI and Gemini stream, Deepgram and ElevenLabs are inline, AssemblyAI is queued), every client is `MediaClient.make(Service, { modality, responseEvents })` (`src/media-client.ts`), which dispatches on the route's `kind`, and every model composes through `composeRoute`. fal queue protocols come from `protocols/utils/fal-queue.ts`, bodies are `json`, `multipart`, or `binary` (a raw upload), and a queued protocol that must upload media before submitting implements `start.prepare` (`MediaProtocol.Prepare`; AssemblyAI `/v2/upload`). - -### URL Construction - -`Endpoint` owns `{ baseURL, path, query }`. Each protocol route includes a canonical endpoint when the provider has one (e.g. `https://api.openai.com/v1`); provider helpers override endpoint fields by configuring the route before selecting a model. Generic OpenAI-compatible routes have no canonical URL and require configuration before execution. - -For providers where the URL is derived from typed inputs (Azure resource name, Bedrock region), the provider helper configures the route endpoint before calling `.model(...)`. Use `AtLeastOne` from `route/auth-options.ts` for inputs that accept either of two derivation paths (Azure: `resourceName` or `baseURL`). - -### Provider Facades - -Provider-facing APIs are configured facades over route values. Endpoint/auth/resource/API-version setup happens before model selection, and model selectors accept only a model or deployment id. Media models use per-modality selectors on the same facade (`openai.image(id)`, `.speech(id)`, `.transcription(id)`, `google.video(id)`) that mirror `openai.responses(id)`; the one-word overlap with the request namespace is accepted over a second construction path: - -```ts -const openai = OpenAI.configure({ apiKey, baseURL }) -const model = openai.responses("gpt-4o-mini") -const image = openai.image("gpt-image-2") - -const azure = Azure.configure({ resourceName, apiKey, apiVersion: "v1" }) -const deployment = azure.responses("my-deployment") - -const gateway = CloudflareAIGateway.configure({ accountId, gatewayId, gatewayApiKey, apiKey }) -const proxied = gateway.model("openai/gpt-4o-mini") -``` - -Keep provider facades small and explicit: - -- Use branded `ProviderID.make(...)` and `ModelID.make(...)` where ids are constructed directly. -- Use `model` for the default API path and named methods for provider-native alternatives such as OpenAI `responses` and `chat`. -- Put provider-specific setup on `.configure(...)`; do not add `model(id, overrides)` as a duplicate construction path. -- Export lower-level `routes` arrays separately only when advanced internal wiring needs them. -- Prefer `apiKey` as provider-specific sugar and `auth` as the explicit override; keep them mutually exclusive in provider option types with `ProviderAuthOption`. -- Resolve `apiKey` → `Auth` with `AuthOptions.bearer(options, "_API_KEY")` (it honors an explicit `auth` override and falls back to `Auth.config(envVar)` so missing keys surface a typed `Authentication` error rather than a runtime crash). -- Use separate top-level facades for products with different required setup, such as `CloudflareAIGateway` and `CloudflareWorkersAI`. -- Give every named provider its own file and top-level export. Keep its endpoint, auth defaults, and route setup in that file. Compose shared protocols directly; do not nest named provider presets under generic compatible facades or keep their endpoints in a shared provider profile registry. - -`Provider.make(...)` remains available for simple static provider definitions, but new built-in providers should prefer plain configured facades unless a helper removes real duplication without adding runtime behavior. - -### Provider Package Entrypoints - -Catalog-selected native providers use package-like export paths from `@opencode/ai`. They are internal entrypoints in one npm package, not separately published provider packages. Every entrypoint implements `ProviderPackage.Definition` and exposes `model(modelID, settings)`, where settings are one flat serializable object: the connection keys the entrypoint declares (`apiKey`, `baseURL`, `region`, …), the common `headers` and `body` overlays, and the protocol's request options (`reasoningEffort`, `thinking`, …) side by side. Each entrypoint destructures its own connection keys and passes the rest to the route as `providerOptions`; there is no nested `providerOptions` at the entrypoint. - -```ts -import { model } from "@opencode/ai/providers/openai/responses" - -const selected = model("gpt-5", { - apiKey, - reasoningEffort: "high", -}) -``` - -Keep semantic APIs as separate entrypoints, such as OpenAI `chat` and `responses`. Transport is execution policy: OpenAI Responses uses HTTP by default and may receive a per-call WebSocket channel executor through `StreamOptions` without changing model or route identity. - -Do not expose `Route` in provider package settings. Route composition stays an implementation detail behind `model(...)`. - -The dependency arrow points down: `providers/*.ts` files import protocol routes and auth-option utilities; protocol modules import `endpoint`, `auth`, `framing`, and transport pieces. Protocols do not import provider facades. Lower-level modules know nothing about provider catalog metadata. `OpenAIResponses` composes the provider-neutral `OpenResponses` protocol; the baseline never imports the OpenAI extension. - -### Shared protocol helpers - -`ProviderShared` exports a small toolkit used inside protocol implementations to keep them focused on provider-native shapes: - -- `joinText(parts)` — joins an array of `TextPart` (or anything with a `.text`) with newlines. Use this anywhere a protocol flattens text content into a single string for a provider field. -- `parseToolInput(route, name, raw)` — Schema-decodes a tool-call argument string with the canonical "Invalid JSON input for `` tool call ``" error message. Treats empty input as `{}`. -- `parseJson(route, raw, message)` — generic JSON-via-Schema decode for non-tool bodies. -- `eventError(route, message, ...)` — typed `InvalidProviderOutput` constructor for stream-time decode failures. -- `validateWith(decoder)` — maps Schema decode errors to `InvalidRequest`. `Route.make(...)` uses this for body validation; lower-level routes can reuse it. -- `matchToolChoice(provider, choice, branches)` — branches over `LLMRequest["toolChoice"]` for provider-specific lowering. - -If you find yourself copying a 3-to-5-line snippet between two protocols, lift it into `ProviderShared` next to these helpers rather than duplicating. - -### Chronological System Updates - -`LLMRequest.system` is the initial privileged prompt that applies ahead of the conversation. `Message.system(...)` is a separate, provider-neutral chronological operator update inside `LLMRequest.messages`; it applies only from its position in history onward and accepts text content only. - -Native chronological system messages are route/model-specific. Open Responses lowers them to standard `developer` messages, while Anthropic Messages lowers them to native system messages for Claude Opus 4.8 (`claude-opus-4-8`). Other routes and models intentionally lower the update in place into ordinary user-compatible text using this stable escaped representation: - -```text - -... - -``` - -The wrapped-user fallback preserves ordering while visibly lowering authority. Never silently pass a raw chronological `role: "system"` through a route that might reject it. Do not insert raw retrieved documents, tool output, or web content into privileged chronological system updates; keep untrusted content in ordinary user/tool channels. - -### Effort Updates - -`Message.effort({ effort, previous })` is a chronological "reasoning effort changed here" marker (`undefined` means the model default). Changing a top-level effort invalidates the whole provider prompt cache, so protocols with a native per-message update (`Protocol.supportsEffortUpdates`) keep the top-level effort at the first marker's `previous` and lower each marker in place: Anthropic Messages emits an empty `role: "system"` message with `output_config.effort` plus the `mid-conversation-output-config-2026-07-01` beta, and OpenAI Responses emits `configuration_update` items. `applyEffortUpdates` runs in `prepareRequest` and strips the markers for every other route, so a protocol without support keeps today's plain top-level behaviour. When the last marker disagrees with the effort the request asks for (reverted or forked history), `resolveEffortUpdates` strips the markers and falls back to a plain top-level change. - -### Tools - -Tool loops are represented in common messages and events: - -```ts -const call = ToolCallPart.make({ id: "call_1", name: "lookup", input: { query: "weather" } }) -const result = Message.tool({ id: "call_1", name: "lookup", result: { forecast: "sunny" } }) - -const followUp = LLM.request({ - model, - messages: [Message.user("Weather?"), Message.assistant([call]), result], -}) -``` - -Routes lower these into provider-native assistant tool-call messages and tool-result messages. Streaming providers should emit `tool-input-delta` events while arguments arrive, then a final `tool-call` event with parsed input. - -### Tool dispatch - -`LLM.stream(request)` and `LLM.generate(request)` each run exactly one model call. Add tool schemas to `request.tools` with `Tool.toDefinitions(tools)`. When a caller wants the package's typed one-call execution behavior, pass each canonical local `tool-call` event to `ToolRuntime.dispatch(tools, call)`. - -```ts -const get_weather = Tool.make({ - description: "Get current weather for a city", - parameters: Schema.Struct({ city: Schema.String }), - success: Schema.Struct({ temperature: Schema.Number, condition: Schema.String }), - execute: (input) => - Effect.gen(function* () { - const data = yield* WeatherApi.fetch(input.city) - return { temperature: data.temp, condition: data.cond } - }), -}) - -const tools = { get_weather, get_time, ... } -const events = yield* LLM.stream( - LLMRequest.update(request, { tools: Tool.toDefinitions(tools) }), -).pipe(Stream.runCollect) - -const call = Array.from(events).find(LLMEvent.is.toolCall) -if (call && !call.providerExecuted) { - const dispatched = yield* ToolRuntime.dispatch(tools, call) - // Persist call + dispatched.result, then construct the next request explicitly. -} -``` - -The dispatcher: - -- On `tool-call`: looks up the named tool, decodes input against `parameters` Schema, dispatches to the typed `execute`, encodes the result against `success` Schema, and returns canonical `tool-result` events. -- Does not stream providers, construct Session events, schedule fibers, append history, count steps, or continue model rounds. -- Leaves persistence and continuation to the enclosing product flow. - -Handler dependencies (services, permissions, plugin hooks, abort handling) are closed over by the consumer at tool-construction time. Build the tools record inside an `Effect.gen` once and reuse it across many dispatches. - -Errors must be expressed as `ToolFailure`. The runtime catches it and emits a `tool-error` event, then a `tool-result` of `type: "error"`, so the model can self-correct on the next step. Anything that is not a `ToolFailure` is treated as a defect and fails the stream. Three recoverable error paths produce `tool-error` events: - -- The model called an unknown tool name. -- Input failed the `parameters` Schema. -- The handler returned a `ToolFailure`. - -Provider-defined / hosted tools (Anthropic `web_search` / `code_execution` / `web_fetch`, OpenAI Responses `web_search_call` / `file_search_call` / `code_interpreter_call` / `mcp_call` / `image_generation_call` / `computer_use_call`) pass through the runtime untouched: - -- Routes surface the model's call as a `tool-call` event with `providerExecuted: true`, and the provider's result as a matching `tool-result` event with `providerExecuted: true`. -- Callers detect `providerExecuted` on `tool-call` and **skip local dispatch** — no handler is invoked and no `tool-error` is raised for "unknown tool". The provider already executed it. -- Callers that continue should retain both events in explicit history when the protocol requires it. Anthropic encodes them back as `server_tool_use` + `web_search_tool_result` (or `code_execution_tool_result` / `web_fetch_tool_result`) blocks; OpenAI Responses callers typically use `previous_response_id` instead of resending hosted-tool items. - -Add provider-defined tools to `request.tools` (no runtime entry needed). The matching route must know how to lower the tool definition into the provider-native shape; right now Anthropic accepts `web_search` / `code_execution` / `web_fetch` and OpenAI Responses accepts the hosted tool names listed above. - -## Protocol File Style - -Protocol files should look self-similar. Provider quirks belong behind named helpers so a new route can be reviewed by comparing the same sections across files. - -### Section order - -Use this order for every protocol module: - -1. Public model input -2. Request body schema -3. Streaming event schema -4. Parser state -5. Request body construction (`fromRequest`) -6. Stream parsing (`step` and per-event handlers) -7. Protocol and route -8. Protocol route export - -### Rules - -- Keep protocol files focused on the protocol. Move provider-specific projection, signing, media normalization, or other bulky transformations into `src/protocols/utils/*`. -- Send `tool.inputSchema` as given. `prepareRequest` applies the tool schema rules (`ToolSchemaProjection.tools`) once per request, including tools in namespaces. A protocol whose API needs a model family's rules for every model declares `sanitizer` instead of transforming schemas itself. -- Use `Effect.fn("Provider.fromRequest")` for request body construction entrypoints. Use `Effect.fn(...)` for event handlers that yield effects; keep purely synchronous handlers as plain functions returning a `StepResult` that the dispatcher lifts via `Effect.succeed(...)`. -- Parser state owns terminal information. The state machine records finish reason, usage, and pending tool calls; emit one terminal `finish` event (or `provider-error`) for each completed response. If a provider splits reason and usage across events, merge them in parser state before flushing. -- Emit exactly one terminal `finish` event for a completed response, normally after a matching `step-finish`. Use `stream.terminal` to stop reading when the provider has a completion sentinel; use `stream.onHalt` when the final event must be flushed after the framed stream ends. -- Use shared helpers for repeated protocol policy such as text joining, usage totals, JSON parsing, and tool-call accumulation. `ToolStream` (`protocols/utils/tool-stream.ts`) accumulates streamed tool-call arguments uniformly. -- Make intentional provider differences explicit in helper names or comments. If two protocol files differ visually, the reason should be obvious from the names. -- Prefer dispatched per-event handlers (`onMessageStart`, `onContentBlockDelta`, ...) called from a small top-level `step` switch over a long if-chain. The dispatcher keeps the event surface visible at a glance. -- Keep tests in the same conceptual order as the protocol: basic prepare, tools prepare, unsupported lowering, text/usage parsing, tool streaming, finish reasons, provider errors. - -### Review checklist - -- Can the file be skimmed side-by-side with `openai-chat.ts` without hunting for equivalent sections? -- Are provider quirks named, isolated, and covered by focused tests? -- Does request body construction validate unsupported common content at the protocol boundary? -- Does stream parsing emit stable common events without leaking provider event order to callers? -- Does `toolChoice: "none"` behavior read as intentional? - -## Recording Tests - -Recorded tests use one cassette file per scenario. A cassette holds an ordered array of `{ request, response }` interactions, so multi-step flows (tool loops, retries, polling) record into a single file. Use `recordedTests({ prefix, requires })` and let the helper derive cassette names from test names: - -```ts -const recorded = recordedTests({ prefix: "openai-chat", requires: ["OPENAI_API_KEY"] }) - -recorded.effect("streams text", () => - Effect.gen(function* () { - // test body - }), -) -``` - -Replay is the default. `RECORD=true` records fresh cassettes locally and requires the listed env vars; unset `CI` before recording because CI always forces replay. Cassettes are written as pretty-printed JSON so multi-interaction diffs stay reviewable. - -Pass `provider`, `protocol`, and optional `tags` to `recordedTests(...)` / `recorded.effect.with(...)` so cassettes carry searchable metadata. Use recorded-test filters to replay or record a narrow subset without rewriting a whole file: - -- `RECORDED_PROVIDER=openai` matches tests tagged with `provider:openai`; comma-separated values are allowed. -- `RECORDED_PREFIX=openai-chat` matches cassette groups by `recordedTests({ prefix })`; comma-separated values are allowed. -- `RECORDED_TAGS=tool` requires all listed tags to be present, e.g. `RECORDED_TAGS=provider:togetherai,tool`. -- `RECORDED_TEST="streams text"` matches by test name, kebab-case test id, or cassette path. - -Filters apply in replay and record mode. Combine them with `RECORD=true` when refreshing only one provider or scenario. - -**Binary response bodies.** Most providers stream text (SSE, JSON). The recorder treats known textual media types (`text/*`, JSON/XML structured types, JavaScript, forms, YAML, and SVG) as text and stores every other response as base64 with `bodyEncoding: "base64"`. This preserves binary formats such as AWS event-stream frames without a lossy UTF-8 round trip. - -**Matching strategy.** A runtime request atomically claims the first unused recorded interaction that matches its method, URL, allow-listed headers, and canonical JSON body. Distinct requests may replay in any order or concurrently. Repeated identical requests consume their matching responses in cassette order, preserving deterministic retry and polling behavior. `scriptedResponses` (in `test/lib/http.ts`) is the deterministic counterpart for tests that don't need a live provider; it scripts response bodies in order without reading from disk. - -Do not blanket re-record an entire test file when adding one cassette. `RECORD=true` rewrites every recorded case that runs, and provider streams contain volatile IDs, timestamps, fingerprints, and obfuscation fields. Prefer deleting the one cassette you intend to refresh, or run a focused test pattern that only registers the scenario you want to record. Keep stable existing cassettes unchanged unless their request shape or expected behavior changed. +- Use `testEffect(...)` from `test/lib/effect.ts` for tests that need Effect layers. +- Provider tests replay cassettes: recorded HTTP exchanges stored as JSON in `test/fixtures/recordings/`, one file per scenario. Replay is the default and needs no API keys. +- To record, unset `CI` and run with `RECORD=true` plus the provider's API key. Re-record only the cassette you are changing; recordings contain volatile IDs that create noisy diffs. diff --git a/packages/ai/docs/media-design.md b/packages/ai/docs/media-design.md deleted file mode 100644 index e1909d595877..000000000000 --- a/packages/ai/docs/media-design.md +++ /dev/null @@ -1,484 +0,0 @@ -# Media generation in `@opencode/ai` — public API direction - -Status: phases 1–4 implemented (through Image queued routes and partial images); phase 5 proposal. - -## Goal - -`@opencode/ai` becomes the one package you reach for to generate anything: text, images, video, speech, transcripts, and later music and realtime. The LLM surface already exists and is shaped by three constraints: Effect-first, used by OpenCode Core, usable externally. Media has a different priority order: **external DX first**, Effect and Promise as peers, Core as one consumer among many. - -The design below is derived from a survey of the raw provider APIs (OpenAI, Gemini/Veo/Imagen, xAI, Stability, BFL, fal, Replicate, Runway, Luma, Kling, MiniMax, ElevenLabs, Deepgram, Cartesia, AssemblyAI, Lyria) and of existing multi-provider SDKs. - -## What the survey forces - -1. **Three execution shapes, everywhere.** Inline sync (OpenAI images, all TTS, Gemini), async job with polling or webhook (every video provider, BFL, fal, Replicate, AssemblyAI), and bidirectional streams (ElevenLabs/Cartesia/Deepgram WS, realtime). Video has no sync provider at all. -2. **Output is never just bytes.** base64, signed URLs with TTLs from 10 minutes (BFL, so its route downloads before returning via `PollContext.materialize`) to 2 days (Veo), URLs that need auth plus redirect (Veo), separate download endpoints (Sora `/content?variant=`), raw bodies (Stability, TTS). Multi-output is the norm. -3. **Inputs have roles.** First/last frame, mask, style/subject reference, source video for edit/extend, reference audio, prior generation id, provider-side file handles (`file_id`, `gs://`, `runway://`, `mm_file://`). -4. **Partial streaming is modality-specific.** Images: a few whole partial frames. Audio: ordered chunks plus timestamp events. Jobs: status/progress/logs. Video: none. -5. **Usage is a union**: tokens, seconds, characters (often only in headers), credits, compute time. -6. **Moderation can be partial success** (Veo strips audio but returns video). Deprecations are constant (Sora API shuts down 2026-09-24; Imagen is shut down on the Gemini API and past its 2026-06-30 discontinuation date on Vertex). - -## Where existing SDKs are weak and we should not be - -- No streaming TTS. -- Video handles are experimental start/status pairs; the polling loop lives inside the generate call. -- Unsupported inputs become silent warnings arrays, so a request can succeed while dropping your mask. -- `n` is fanned out into hidden parallel calls, which obscures cost and idempotency. -- Each modality has its own bespoke result type; the file abstraction is a lazy base64/bytes pair with no URL, expiry, or provider ref. -- Effect's own `unstable/ai` has no media generation. Nothing in the Effect ecosystem owns this. - -## Design principles - -- **Same shape as LLM.** `X.request(...)` → Schema class; `X.generate(request)` / `X.stream(request)`; `XClient.Service` + `layer`; typed `AIError`. If you know `LLM`, you know `Video`. -- **Execution shape is route policy, not API shape.** `Image.generate` returns an image whether the provider is inline or queued. Job control is available uniformly when you want it. -- **Errors, not warnings.** Unsupported common fields fail at the protocol boundary with a typed `AIError`, as the LLM routes do today. Provider-side partial results (filtered audio, moderated sample) surface as `notices` on the response, never as silent drops. -- **One asset type in, one asset type out**, shared with LLM messages and tool results. -- **Typed per-model options**, no hidden fan-out, no implicit retries that spend money. -- **Promise API is one mechanism for the whole package**, not a media-only wrapper. -- **One construction path per model.** Media models come from per-modality selectors on the configured facade (`openai.image("gpt-image-2")`), the same shape as `openai.responses("gpt-5")`. - -## Public API - -### Model selection - -A model value is built as `OpenAI.configure({ apiKey }).responses("gpt-5")` or `.image("gpt-image-2")`: `configure` fixes credentials, endpoint, and defaults; the selector fixes which of the provider's APIs to hit and binds the typed `providerOptions` generic. Media follows the same shape with one selector per modality — `.image(id)`, `.video(id)`, `.speech(id)`, `.transcription(id)` on the facades that offer each — mirroring `openai.responses(id)`. `Image.request` accepts `ImageModel` only, exactly as `LLM.request` accepts `LanguageModel`. - -```ts -import { OpenAI, Google } from "@opencode/ai/providers" - -const openai = OpenAI.configure({ apiKey }) // OpenAI(...) alone uses env auth (OPENAI_API_KEY) - -LLM.request({ model: openai.responses("gpt-5"), prompt }) -Image.request({ model: openai.image("gpt-image-2"), prompt }) -Video.request({ model: google.video("veo-3.1-generate-preview"), prompt }) -Speech.request({ model: openai.speech("gpt-4o-mini-tts"), text }) -Transcription.request({ model: openai.transcription("gpt-4o-transcribe"), audio }) -``` - -The request namespace and the selector share one word (`Image.request` + `.image(...)`). That redundancy is accepted: a callable facade returning a lazily resolved ref would be a second way to construct the same model, and the type machinery to infer `providerOptions` through it is not worth one word. Where a provider has two routes for one modality, the selectors stay explicit (`openai.chat`, `stability.image` inline vs `stability.upscale()` queued), and one default per modality per provider is part of the facade definition (OpenAI image → Images API, Google image → Gemini-native; Imagen is shut down, so there is no `google.imagen`). The facade selector (`openai.image(id)`) is the public path for media models. Modality-specific package entrypoints (`model(modelID, settings)` beside today's LLM paths such as `@opencode/ai/providers/openai/responses`) are deferred until Core has a modality-aware model resolver; Core's resolver accepts only `LanguageModel` today. - -### `Media` — the asset type - -Replaces `MediaPart.data: string | Uint8Array`, `ImageInput`, `GeneratedImage`, and aligns `Tool.FileContent`. - -```ts -import { Media } from "@opencode/ai" - -Media.Source = - | { type: "bytes"; data: Uint8Array; mediaType: string } - | { type: "base64"; data: string; mediaType: string } - | { type: "url"; url: string; mediaType?: string; expiresAt?: number } - | { type: "ref"; provider: ProviderID; id: string; mediaType?: string } // file_id, gs://, runway://, prior generation - -class Media.Asset { - readonly source: Media.Source - readonly mediaType: string // always resolved (sniffed when the provider omits it) - readonly kind: "image" | "video" | "audio" | "document" | "other" - readonly info?: { width?; height?; durationSeconds?; sampleRate?; channels?; encoding?; format? } - readonly expiresAt?: number - readonly providerMetadata?: ProviderMetadata - readonly headers?: Record // transient download credentials (Veo); never in source/JSON - - bytes(): Effect // downloads/decodes lazily, cached - base64(): Effect - dataUrl(): Effect - materialize(): Effect // url/ref → bytes, before the URL dies -} - -Media.bytes(data, mediaType?) Media.base64(data, mediaType?) -Media.url(url, options?) Media.ref(provider, id) -Media.file(path) // Effect: reads + sniffs -Media.write(asset, path) // Effect -``` - -`Media.file` and `Media.write` stay Effect-only: bring your platform's `FileSystem` layer. The Promise client owns the -runtime path: `ai.file(path)` and `ai.write(asset, path)` read and write through `node:fs/promises` (loaded on first -use) with the same media-type sniffing and `InvalidRequest` failures, and `ai.bytes`, `ai.base64`, and -`ai.materialize` run the asset methods in its runtime. - -A `ref` source is accepted as input only by routes whose provider issues file handles. No shipped route produces one -yet, so `bytes()` and `materialize()` on a ref fail by design until a producer exists. - -Raw-PCM outputs (Gemini TTS, Cartesia raw, Deepgram WS) carry `info.encoding/sampleRate/channels` because there is no container header. - -### Modality namespaces - -Each namespace mirrors `LLM` exactly. - -```ts -import { Image, Video, Speech, Transcription } from "@opencode/ai" -import { OpenAI, Google, ElevenLabs, Fal } from "@opencode/ai/providers" -``` - -#### Image - -```ts -Effect.gen(function* () { - const request = Image.request({ - model: openai.image("gpt-image-2"), - prompt: "A robot tending a rooftop garden", - images: [yield* Media.file("./ref.png")], // references / edit sources - mask: yield* Media.file("./mask.png"), - n: 2, - size: "1536x1024", // OpenAI sizes by pixels; Gemini/xAI take aspectRatio instead - format: "webp", - providerOptions: { quality: "high", background: "transparent" }, // typed per model - }) - - const response = yield* Image.generate(request) // ImageResponse - response.image // Media.Asset (first) - response.images // Media.Asset[] - response.usage // Usage union (see below) - response.notices // moderation / partial-result notices - - Image.stream(request) // Stream - // ImageEvent: generation-queued | generation-progress | image-partial { index, image } | image { index, image } | finish { usage } -}) -``` - -`size` and `aspectRatio` are not interchangeable; each route rejects fields it cannot lower — see the portability table -in the README's Image generation section. - -Editing is not a separate function; `images`/`mask` on the request select the edit path in the route (OpenAI `/images/edits`, Gemini multimodal parts, xAI `/images/edits`). Routes that cannot honor `mask` fail with `Unsupported`. - -`ImageRoute` is the inline | stream | queued union, dispatched on `route.kind`, like every modality route. `Image.stream` on a streaming route emits `image-partial` previews before each `image`; on a queued route it emits `generation-queued` / `generation-progress` observations, then the result's `image` and `finish` events. - -#### Video - -Shipped in phase 2 (`src/video.ts`, `src/video-client.ts`, protocols `google-video`, `xai-video`, `fal-video`, `runway-video`). - -```ts -Effect.gen(function* () { - const request = Video.request({ - model: google.video("veo-3.1-generate-preview"), - prompt: "Panning wide shot of a calico kitten sleeping in the sunshine", - frames: { first: yield* Media.file("./start.png"), last: yield* Media.file("./end.png") }, - references: [yield* Media.file("./style.png")], - video: Media.bytes(previous, "video/mp4"), // edit / extend source - durationSeconds: 8, - aspectRatio: "16:9", - resolution: "1080p", - audio: true, - n: 1, - seed: 7, - negativePrompt: "text, watermark", // common, not provider-native - providerOptions: { personGeneration: "allow_adult" }, - }) - - // Simple: wait for it. - const response = yield* Video.generate(request, { poll: { interval: "10 seconds", timeout: "10 minutes" } }) - response.video // Media.Asset: url (expiresAt on Veo and Runway; transient `headers` for Veo downloads) - response.usage // credits on Runway; the other three report none (xAI's usage.cost_in_usd_ticks is not decoded) - response.notices // Veo raiMediaFilteredReasons → filtered, xAI respect_moderation → moderated - yield* response.video.materialize() // pull bytes before the URL expires - - // Explicit generation control. - const generation = yield* Video.start(request) // Generation - generation.id; generation.status; generation.progress; generation.position; generation.token - yield* generation.await({ poll }) // VideoResponse - yield* generation.cancel() // fal PUT cancel_url, Runway DELETE /tasks/{id}; Veo and xAI succeed without a request - - // Resume from another process. The token is validated against the route's codec and refreshed once. It carries no - // route identity, so persist the provider and model ID alongside it: `resume` needs the model. - const resumed = yield* Video.resume(model, JSON.parse(saved)) - - // Progress as a stream. - Video.stream(request, { poll }) // Stream: generation-queued { id, position } | generation-progress { id, progress } | video { index, video } | finish { usage, notices } -}) -``` - -Tokens are route-owned JSON: Veo `{ operation }`, xAI `{ requestID }`, Runway `{ taskID }`, fal -`{ requestID, statusURL, responseURL, cancelURL }` (fal's follow-up URLs are authoritative and absolute). Common-field -lowering per provider: Veo takes inline media only, rejects `audio: false` and `n > 1`, and requires `frames.first` -when `frames.last` is set; xAI rejects `n`, `seed`, and `negativePrompt` and routes a `video` input to edits or -(`providerOptions.mode: "extend"`) extensions; fal rejects `n`, plus `durationSeconds`, `references`, and `frames.last` -because the field names and enums differ per model; Runway passes `aspectRatio` through as its pixel `ratio` and -rejects `n`. - -Deferred: `Video.complete(model, token, webhook)` (finish from a webhook payload without polling) and provider poll -hints (none of the four providers emit one). Later providers: Luma, Kling, MiniMax, Replicate. - -#### Speech (TTS) - -Shipped in phase 3 (`src/speech.ts`, `src/speech-client.ts`, protocols `openai-speech`, `google-speech`, -`elevenlabs-speech`, `cartesia-speech`, `deepgram-speech`; new `ElevenLabs`, `Cartesia`, and `Deepgram` facades). - -```ts -const request = Speech.request({ - model: elevenlabs.speech("eleven_flash_v2_5"), - text: "Hello from OpenCode.", - voice: "JBFqnCBsd6RMkjVDRZzb", // provider-native identifier, or { id } - format: "mp3", // mp3 | wav | pcm | opus | aac | flac | (string & {}) - speed: 1.0, - language: "en", - instructions: "Warm, unhurried.", // only OpenAI; elsewhere fails typed - timestamps: true, // request alignment; routes without it fail typed - providerOptions: { voice_settings: { stability: 0.5 } }, -}) - -const response = yield* Speech.generate(request) // SpeechResponse: audio: Media.Asset, timestamps?, usage?, providerMetadata? -yield* Speech.stream(request) // Stream: generation-queued | generation-progress | audio-delta { chunk } | timestamps { items } | finish { audio, usage? } -``` - -Execution is `MediaProtocol.stream` for every provider: one request whose body is framed and folded by a `step` -state machine, with `generate` running the same stream and collecting it. The route submits the request with its -`mode` (`"generate" | "stream"`), which lets one provider stay one protocol — OpenAI adds `stream_format: "sse"` (except `tts-1`/`tts-1-hd`, which stream raw bytes), ElevenLabs appends -`/stream`, Cartesia switches `/tts/bytes` to `/tts/sse`, Gemini switches `generateContent` to -`streamGenerateContent`. The terminal `finish` event carries the assembled asset (every provider's stream is -concatenable chunks), so stream consumers also get the whole file and `generate` is just "take `finish`, gather -`timestamps`". The cost is memory: a stream holds every chunk until `finish`, so even a consumer that only plays deltas -keeps the whole clip in memory. That is bounded by the providers' input text limits (a few minutes of audio); a -long-form or session API would need an opt-out. - -**Voice.** `voice?: string | { id: string }`. A string is passed through as the provider's native identifier — a -name on OpenAI and Gemini, a voice id on ElevenLabs (path segment) and Cartesia. `{ id }` selects an OpenAI custom -voice and is treated as the plain string on routes that do not distinguish custom from built-in. Deepgram's voice is -the model id (`aura-2-thalia-en`), so `voice` is `unsupported` there. There is no cross-provider voice catalog or -name→id resolution. Multi-speaker (Gemini `speechConfig.multiSpeakerVoiceConfig`) and per-voice settings -(ElevenLabs `voice_settings`) go through `providerOptions`. - -**Format and PCM.** `format` is container-level; provider sample rates and bitrates live under `providerOptions` -(ElevenLabs `outputFormat`, Cartesia `sampleRate`/`bitRate`/`encoding`, Deepgram `encoding`/`container`/`sampleRate`/ -`bitRate`). Each protocol maps `format` to its native value (ElevenLabs `mp3_44100_128`/`pcm_24000`/`wav_24000`/ -`opus_48000_64`, Cartesia `{ container, encoding, sample_rate }`, Deepgram `encoding`+`container`) and declares the -asset's media type rather than sniffing, because headerless PCM can look like an MPEG frame sync. Headerless PCM -always carries `info.encoding`, `info.sampleRate`, and `info.channels`; its media type is the provider's declaration -(Gemini `audio/L16;codec=pcm;rate=24000`, Deepgram's `content-type`) or `audio/pcm`. Gemini's asset follows the -provider's declared type: WAV for Gemini 3.8 TTS `generate`, headerless PCM otherwise. The route never wraps PCM as WAV, -so `pcm` is the only explicit `format` it accepts, and not on Gemini 3.8 `generate`. Every `format` value a route cannot -produce (unknown to it, a container on Cartesia SSE, WAV on an ElevenLabs stream, anything but `pcm` on Gemini, `pcm` on -Gemini 3.8 `generate`) fails the same way as an unsupported field: `UnsupportedOperation` with -`operation: "media.format"`. - -**Timestamps.** `timestamps: true` on the request asks for alignment. ElevenLabs selects the `with-timestamps` -endpoints (character-level, NDJSON when streaming); Cartesia sets `add_timestamps` on `/tts/sse` (word-level; a -`generate` with timestamps collects the SSE stream). OpenAI, Gemini, and Deepgram reject it. - -Common-field lowering per provider: - -| Provider | `voice` | `speed` | `language` | `instructions` | `timestamps` | Usage | -|---|---|---|---|---|---|---| -| OpenAI | `voice` (name or `{ id }`) | `speed` | unsupported | `instructions` | unsupported | `tokens` from SSE `speech.audio.done` only | -| Gemini | `prebuiltVoiceConfig.voiceName` | unsupported | `speechConfig.languageCode` | unsupported (direct in text) | unsupported | `tokens` from `usageMetadata` | -| ElevenLabs | path voice id (required) | `voice_settings.speed` | `language_code` | unsupported | `with-timestamps` | `credits` from `character-cost` header | -| Cartesia | `voice` (required) | `generation_config.speed` | `language` | unsupported | `add_timestamps` | none | -| Deepgram | unsupported (voice is the model) | `speed` query | unsupported | unsupported | unsupported | `characters` from `dg-char-count` header | - -Deferred: `Speech.session(...)` — input-streaming TTS where text arrives incrementally over a WebSocket (ElevenLabs -`stream-input`, Cartesia WebSocket contexts, Deepgram WebSocket speak) — is a separate scoped resource, not part of -`generate`/`stream`, and ships with the realtime work in phase 5. - -#### Transcription (STT) - -Shipped as the second half of phase 3 (`src/transcription.ts`, `src/transcription-client.ts`, protocols -`openai-transcription`, `google-transcription`, `deepgram-transcription`, `elevenlabs-transcription`, -`assemblyai-transcription`; new `AssemblyAI` facade). - -```ts -const request = Transcription.request({ - model: openai.transcription("gpt-4o-transcribe-diarize"), - audio: yield* Media.file("./call.wav"), - language: "en", // provider-native passthrough - timestamps: "segment", // none | segment | word - diarize: true, - speakers: 2, // speaker count (AssemblyAI exact, ElevenLabs maximum) - providerOptions: { known_speaker_names: ["agent"] }, -}) - -const response = yield* Transcription.generate(request) -response.text; response.segments; response.words; response.language; response.durationSeconds; response.usage -yield* Transcription.stream(request) // Stream: generation-queued | generation-progress | text-delta | segment | finish -const generation = yield* Transcription.start(request) // queued routes only -yield* Transcription.resume(model, token) -``` - -Transcription is the first modality whose providers span all three protocol kinds, and it needed no fourth kind. -Every `MediaRoute` now carries its `kind`; `TranscriptionRoute` is the union of the inline, stream, and queued routes; -`TranscriptionModel.fromRoute` is overloaded per protocol kind (arity picks the overload: ``, -``, ``) and composes through the shared `composeRoute` (`src/media-model.ts`), -which picks `MediaRoute.inline` / `stream` / `queued`; and `TranscriptionClient`, like every modality client, is -`MediaClient.make` (`src/media-client.ts`), which dispatches on `route.kind`. `generate` on a queued route is `start` -then `await`; `stream` on an inline route is the response as a single `finish`, and on a queued route it is the status -observations followed by `finish`. `start` / `resume` on a non-queued route fail with `UnsupportedOperation` (`transcription.start`). The -`finish` event carries the whole transcript (text, segments, words, language, duration, usage), so the stream route's -`collect` is just "take `finish`". - -The route layer gained a `binary` body with array-valued `query` (Deepgram) and `Queued.start.prepare` (AssemblyAI's -upload); `packages/ai/AGENTS.md` (Media Routes) describes both. - -Settled rules: - -- **Timestamps.** A granularity the selected route or model cannot produce fails as `UnsupportedOperation` - (`media.timestamps`), following Speech; a route that returns more than asked (Deepgram, ElevenLabs, and AssemblyAI - always return words) is not stripped. Segments always carry start and end times: Gemini times each transcription part from its - word offsets, so segment timestamps and diarization also request word offsets there. -- **Diarization.** `diarize` means segments (and words, where the provider labels them) carry `speaker`. Labels are - provider-native strings — OpenAI `A` or a known speaker name, Deepgram `0`, Gemini `spk:0`, AssemblyAI `A`, - ElevenLabs `speaker_0` — with no cross-provider speaker model. `speakers` is the number of speakers to label: - AssemblyAI (`speakers_expected`) treats it as an exact constraint rather than a hint, and ElevenLabs - (`num_speakers`) as the maximum. Both turn on diarization for it; the other routes reject it. -- **Segments from words.** ElevenLabs returns only a token list (`word`, `spacing`, `audio_event`), so its segments - are speaker turns: consecutive words and spacing with one `speaker_id`, text joined from the provider's own spacing - tokens. `words` drops spacing and audio events. Segments therefore need diarization, which `timestamps: "segment"` - turns on, as AssemblyAI's utterances need speaker labels. -- **Language** is passed through (`language`, OpenAI `gpt-transcribe` `languages[]`, Gemini `languageCodes`, - AssemblyAI and ElevenLabs `language_code`). `response.language` is the provider's own value, lowercased but not - normalized: an ISO code on most routes (AssemblyAI's detection returns `en`, ElevenLabs ISO 639-3 `eng`), `english` - from whisper-1. Deepgram and AssemblyAI assume English unless asked to detect, so a missing `language` enables their - detection. -- **Gemini** requires a transcribe model; other model ids fail with `UnsupportedOperation` before the call, because - general models ignore `audioTranscriptionConfig` and answer conversationally. Streamed chunks carry whole speaker - turns (one part per turn), which join with a space. -- **Streaming inline providers** emit only `finish`; deltas are never faked. -- **Units.** AssemblyAI milliseconds and Gemini protobuf durations (`"0.400s"`) are normalized to seconds at the - protocol boundary. - -| Provider | Kind | Audio input | `timestamps` | `diarize` | Unsupported | Usage | -|---|---|---|---|---|---|---| -| OpenAI | stream (`stream: true` in `stream` mode; `whisper-1` ignores `stream`, so it emits only `finish`) | multipart `file` (inline only) | `whisper-1` (`verbose_json`); diarize model: `segment` | `gpt-4o-transcribe-diarize` (`diarized_json`) | `speakers`; `prompt` on the diarize model | `tokens` or `seconds` | -| Gemini | stream (`generateContent` / `streamGenerateContent`) | `inlineData` or Gemini Files `fileData` | `audioTranscriptionConfig.wordTimestamp` | `audioTranscriptionConfig.diarization` | `prompt`, `speakers` | `tokens` | -| Deepgram | inline | raw body, or JSON `{ url }` | words always; `segment` → `utterances` | `diarize_model=latest` + `utterances` | `prompt`, `speakers` | `seconds` (`metadata.duration`) | -| ElevenLabs | inline | multipart `file`, or `source_url` | words always; `segment` → `diarize` (speaker turns) | `diarize` | `prompt`; `webhook`, per-channel `use_multi_channel` | `seconds` (`audio_duration_secs`) | -| AssemblyAI | queued (upload → submit → poll) | `/v2/upload` then `audio_url`, or a URL | words always; `segment` → `speaker_labels` | `speaker_labels` | — | `seconds` (`audio_duration`) | - -Deferred: `Transcription.session(...)` — realtime STT over WebSocket (Deepgram live, AssemblyAI streaming, ElevenLabs -realtime, OpenAI realtime transcription) — is the same future scoped `session` shape as input-streaming TTS and ships -with the realtime work in phase 5. - -### `Generation` — shared async execution - -```ts -class Generation { - readonly id: string - readonly route: GenerationRoute // token-free: { status, result, cancel?: Effect } closed over the decoded token - readonly token: unknown // route-owned serializable JSON - readonly status: "queued" | "running" | "completed" | "failed" | "cancelled" | "expired" - readonly progress?: number // 0..1, normalized - readonly position?: number - refresh(): Effect, AIError> - result(): Effect - await(options?: GenerationAwaitOptions): Effect - cancel(): Effect - events(options?: GenerationAwaitOptions): Stream // fails with Timeout past poll.timeout, checked per observation -} - -GenerationAwaitOptions = { poll?: Poll } -Poll = { interval?: Duration; timeout?: Duration } -``` - -`Generation` is not video-specific. Image routes on BFL, fal, Replicate, and Stability `upscale()` are queued; `Image.start` exists for them. A route declares itself `inline` or `queued`; `generate` on a queued route is `start` then `await`. - -Status polls and result reads retry transient failures (rate limits, provider 5xx, and transport errors, classified by the same `isRetryable` the Session runner uses) inside `MediaRoute.queued`. Only the HTTP exchange retries, never the decoded document: a terminal `failed` generation also surfaces as `ProviderInternal` and must not be re-read. Gaps grow exponentially from 1s with jitter, up to 30s each, honoring a provider `retry-after` up to that cap, for at most 8 retries. `await`, `events`, and `Video.stream` cut retries off at `poll.timeout` and fail with `Timeout`, so retries never extend the caller's deadline; a direct `result()` or `resume` read is bounded by the retry cap alone. `start` and `cancel` never retry: a repeated submit can start and bill a second job. The policy is internal; there is no option for it. - -Interrupting `await`, `events`, or `Video.stream` (or aborting the promise API's `signal`) stops waiting only. The provider job keeps running and billing; call `cancel()` explicitly to stop it. - -### Usage - -```ts -Usage = - | { type: "tokens"; input; output; total; details? } - | { type: "seconds"; seconds } - | { type: "characters"; characters } - | { type: "credits"; credits } - | { type: "compute"; seconds } -``` - -Header-only usage (ElevenLabs `character-cost`, Deepgram `dg-char-count`) is lifted into `usage` by the route. - -### Promise API — `@opencode/ai/promise` - -Mirrors the `packages/plugin/src/effect` and `packages/plugin/src/promise` split that already exists in this repo. One mechanism for LLM and media. - -```ts -import { AI } from "@opencode/ai/promise" - -const ai = AI.make() // ManagedRuntime over RequestExecutor.fetchLayer + all clients -// AI.make({ layer }) to inject a custom executor / recorder / middleware - -const image = await ai.image.generate({ model, prompt }) -await ai.bytes(image.image) // also ai.base64, ai.materialize, ai.write(asset, path) -const reference = await ai.file("./ref.png") - -for await (const event of ai.speech.stream({ model, text, voice })) { … } - -const generation = await ai.video.start({ model, prompt }) // snapshot handle; refresh() returns a new one -for await (const event of generation.events({ poll: { interval: 10_000 } })) { … } -const video = await generation.await({ poll: { interval: 10_000 }, signal }) -const resumed = await ai.video.resume(model, JSON.parse(saved)) // persist provider + model ID with the token - -const request = ai.llm.request({ model, prompt }) -const text = await ai.llm.generate(request) -for await (const event of ai.llm.stream(request)) { … } - -await ai.dispose() -``` - -Streams become `AsyncIterable` via `Stream.toAsyncIterable`. `AIError` is thrown as-is. Aborting an `AbortSignal` interrupts the work and, like `fetch`, rejects the Promise or throws from the stream with `signal.reason` instead of ending the stream as if complete. Nothing in `src/*` except this entrypoint knows about promises. - -### Providers - -Existing facades gain per-modality selectors; the modality routes each facade provides (*italics* are not -implemented): - -| Facade | llm | image | video | speech | transcription | other | -|---|---|---|---|---|---|---| -| `OpenAI` | responses (default), chat | Images API (stream) | *Sora skipped (decision 8)* | ✓ | ✓ | | -| `Google` | Gemini | Gemini-native | Veo | Gemini TTS | `gemini-3.5-transcribe` | | -| `XAI` | ✓ | ✓ | ✓ | | | | -| `ElevenLabs` | | | | ✓ | Scribe | *soundEffect, music (phase 5)* | -| `Cartesia` | | | | ✓ | | | -| `Deepgram` | | | | Aura | ✓ | | -| `Fal` | | ✓ (queued) | ✓ | | | | -| `AssemblyAI` | | | | | ✓ (queued) | | -| `BlackForestLabs` | | ✓ (queued) | | | | | -| `Replicate` | | ✓ (queued) | | | | | -| `Stability` | | `image` (inline), `upscale()` (queued) | | | | | -| `Runway` | | | ✓ | | | | -| `Luma`, `Kling`, `MiniMax` | | *deferred* | *deferred* | | | | - -New facades follow the existing one-file-per-provider rule. The facade selector is the public path for media models; modality-specific package entrypoints (for example `@opencode/ai/providers/openai/images`) are deferred until Core has a modality-aware model resolver. - -`ImageModel` gives typed `providerOptions` per model; `VideoModel`, `SpeechModel`, and `TranscriptionModel` follow the same generic. As with `LanguageModel`, the route type does not carry `Options`, so `ImageModel` is an `ImageModel` and client methods take plain `ImageRequestFor`. They share an internal `MediaModel` base class (ids, route, `http` overlays) that is not part of the public exports; `Generation` and the promise client work with the concrete modality models. - -### Routes and protocols - -Media does not fit the LLM four-axis route (SSE frames → event state machine) except for streaming TTS/STT. Reuse `Endpoint`, `Auth`, `Framing`, `RequestExecutor`, and add media protocol kinds: - -- `MediaProtocol.inline` — `body.from(request)` (JSON, multipart, or query), `response.decode(response)` (JSON, or binary body → `Media.Asset`). -- `MediaProtocol.queued` — `start` (body + decode to `{ token, snapshot }`), `status`, `result`, optional `cancel`, and a `token` codec. `result` is always a separate GET (against the status document for Veo/xAI/Runway, fal's `response_url` otherwise) so `await` after `start` and after `resume` share one path. `PollContext.auth` hands the auth headers the route sent to the protocol for output URLs that need them (Veo downloads); they become transient `Media.Asset.headers`, never part of `source`. There is no separate `download` step: `Media.Asset.bytes()` downloads through the executor with those headers. `MediaRoute.inline(...)` / `MediaRoute.queued(...)` compose each kind with endpoint and auth; the queued route decodes the token once and hands `Generation` a token-free `{ status, result, cancel? }`. -- `MediaProtocol.stream` — `body.from(request)` over the request plus its `mode`, `frames` (a function that picks the framing for the call: `Framing.sse`, `lines`, `document`, or the raw bytes), fresh per-response `initial()` state, `step` emitting modality events, and `finish(state, context)` — with the observed response for header-only usage — emitting exactly one terminal event or failing as an incomplete stream. The route fills `reason.http` on stream errors. `MediaRoute.stream(...)` exposes `stream` and `generate` (the same stream folded by the modality's `collect`). - -`MediaRoute.inline` / `MediaRoute.queued` / `MediaRoute.stream` compose one protocol kind with endpoint/auth and tag the route with its `kind`; `ImageModel`/`VideoModel`/`SpeechModel`/`TranscriptionModel` share the `MediaModel` base (`src/media-model.ts`). - -### LLM integration - -- `MediaPart` becomes `{ type: "media"; media: Media.Asset; … }` so protocols branch on `kind` and can pass `url`/`ref` sources through natively (OpenAI `image_url`, Gemini `fileData`). -- New `LLMEvent`s: `media { media: Media.Asset }` so Gemini inline image output is first-class instead of dropped. OpenAI Responses `image_generation_call` keeps its single carrier — the provider-executed `tool-result` with `file` content — because Core consumes hosted tool-result content today and has no `media` event handling yet; it switches to the `media` carrier when Core adopts the event, so the image is never emitted twice. -- `Message.assistant([...])` accepts media parts; Gemini multi-turn image editing replays them. -- `Tool.FileContent` aligns with `Media.Source`. - -## Decisions - -All settled: - -1. **Per-modality selectors** (`openai.image(id)`, `.video`, `.speech`, `.transcription`) name media models, mirroring `openai.responses(id)`. The one-word overlap with the request namespace is accepted over a callable-facade `ModelRef` as a second construction path. -2. **`providerOptions` everywhere** (rename current `Image.options`) for consistency with LLM. -3. **No hidden `n` fan-out.** `n` lowers natively; routes that cannot do `n > 1` fail typed. Callers use `Effect.all` / `Promise.all` explicitly. -4. **Errors over warnings** for unsupported common fields; `notices` for provider-side partial results only. -5. **`Media.Asset` is a class** (lazy bytes, cached) with `Media.Source` as the serializable Schema for wire/persistence. `Asset.from(source)` / `asset.source` round-trip losslessly. Same pattern as `LanguageModel` today. -6. **Promise entrypoint**: `@opencode/ai/promise` exporting `AI.make(options?: { layer? })` plus a module-level default `ai` for scripts, covering LLM too. -7. **Modality set for v1**: `Image`, `Video`, `Speech`, `Transcription`. `Music`/`SoundEffect` and `session` (bidirectional WS, realtime) are designed-for but deferred. -8. **Sora is skipped** (API shuts down 2026-09-24). Video launches with Veo, xAI, fal, Runway. - -## Build order - -Foundation + Image ship together as the reference implementation, serially. Video, Speech, and Transcription then proceed in parallel on separate branches. Image jobs and partial streaming come last, after Video has hardened `Generation`. - -## Phasing - -1. **Foundation** — per-modality selectors, `Media`, `Generation`, `Poll`, `Usage` union, `MediaProtocol` kinds, `@opencode/ai/promise` with `llm` + `image`. Port the five existing image protocols onto it. Unify `MediaPart` and add the `media` LLM event (fixes Gemini image output being dropped). -2. **Video** — ✅ Veo, xAI, fal, Runway shipped (`MediaProtocol.queued`, `Video.start/generate/resume/stream`, promise `ai.video`). Deferred: `Video.complete` (webhooks), Luma, Kling, MiniMax, Replicate. -3. **Speech + Transcription** — ✅ Speech: OpenAI, Gemini TTS, ElevenLabs, Cartesia, Deepgram shipped (`MediaProtocol.stream`, `Speech.generate/stream`, promise `ai.speech`). ✅ Transcription: OpenAI, Gemini, Deepgram, ElevenLabs Scribe, AssemblyAI shipped across all three route kinds (`Transcription.generate/stream/start/resume`, promise `ai.transcription`). Deferred: `Speech.session` and `Transcription.session` (WebSocket streaming). -4. **Image queued routes and partials** — ✅ BFL, fal, Replicate, and Stability creative upscale queued; Stability generate inline; OpenAI `partial_images` streaming (`image-partial` restored). Imagen dropped: shut down on the Gemini API and discontinued on Vertex (2026-06-30). Deferred: Stability's synchronous edit and fast/conservative upscale endpoints. -5. **Later** — ElevenLabs music/SFX, Lyria, `Speech.session` / `Transcription.session`, realtime. - -Core adoption (session attachments beyond png/jpeg/gif/webp/pdf, image-generation tool, TUI rendering) comes after phase 1 and is a Core concern. diff --git a/packages/cli/AGENTS.md b/packages/cli/AGENTS.md index a39796a59a19..345180d04621 100644 --- a/packages/cli/AGENTS.md +++ b/packages/cli/AGENTS.md @@ -1,4 +1,4 @@ # CLI and TUI development guide -- Use `@opencode/client` and the location-scoped data in `packages/tui/src/context/data.tsx` instead of adding dependencies on legacy sync state. +- Use `@opencode/client` and the location-scoped data in `packages/tui/src/context/data.tsx`. - Preserve established TUI behavior unless the task intentionally changes it. diff --git a/packages/codemode/AGENTS.md b/packages/codemode/AGENTS.md index 5fbfecb96a28..9a96a9a8f6bd 100644 --- a/packages/codemode/AGENTS.md +++ b/packages/codemode/AGENTS.md @@ -1,27 +1,5 @@ # @opencode/codemode -- This local package owns confined execution over explicit schema-described tools. Applications own authorization, persistence, external authority, and tool-specific delivery semantics. -- Do not add a speculative generic permission or approval policy. A host omits tools it does not expose and enforces domain authorization inside each provided tool. -- Keep Code Mode unaware of host session, channel, and conversation models. The hosting application supplies trusted execution scope around it. -- Tool schemas are the model-facing Interface. Keep arguments minimal and natural to the operation; never add unrelated IDs as ambient capability tokens. -- State model-visible diagnostics, logs, tool descriptions, and instructions directly. The execution context is already clear; do not repeat `Code Mode` or `CodeMode` unless the distinction is necessary. -- When interpreter behavior or support changes, update `interpreter-support.md` and direct tests in the same PR. -- Program values are `Value` (`src/interpreter/objects.ts`); host values are `unknown` and are copied in at the boundaries (`fromHost`, `fromJson`). Do not widen program-facing signatures back to `unknown`. -- A built-in kind of object is one `Obj` subclass (`Wrapper` for host-backed data such as Date or Map, `Opaque` for machinery such as functions and promises) that overrides `tag`, `toString`, `toPrimitive`, `inspect`, `toHost`, and `iterator` as needed. Do not add `instanceof` ladders over the built-in classes elsewhere; ask the object. - -## OpenAPI - -- Generate an operation only when its transport semantics are supported; otherwise return a precise `skipped` reason. -- Never guess parameter serialization or malformed security semantics. Unsupported serialization is skipped and malformed security fails closed. -- Render unresolved schema constructs as `unknown`, never as invented TypeScript names. -- Keep network reads bounded and map expected encoding, transport, and decoding failures to model-safe `ToolError` values. -- Test supported behavior directly; do not reproduce adapter algorithms in tests. - -## Future Design Notes - -- If a captured user-visible output channel returns (an earlier `output.text`/`output.file`/`output.image` API was removed from v1), keep `output` as its name, distinct from the program return value: `return` stays the structured result for the model, while `output.*` describes artifacts the host may render into a conversation or UI after execution. Keep this host-neutral and let applications decide how captured output is delivered. In v1, hosts collect media host-side (outside CodeMode) instead. -- Improve the failure taxonomy. Distinguish parse/compile mistakes, unsupported syntax, user-thrown errors, invalid returned data, tool refusal, tool internal failure, timeout, and genuine runtime defects so agents can recover accurately instead of treating everything as a generic execution failure. -- Report host failure messages and underlying causes rather than replacing them with generic diagnostics. Preserve interruption behavior. -- Think deliberately about richer binary boundaries before allowing `Blob`, `File`, `ArrayBuffer`, streams, or typed arrays beyond today's JSON-like values. If CodeMode supports binary tool args/results, use explicit tagged data shapes and clear size limits rather than relying on ambient runtime serialization. -- Keep host capabilities explicit. Globals such as `fetch`, `crypto`, filesystem handles, extra modules, or network clients should be opt-in runtime capabilities with obvious policy defaults, not ambient authority. Default to unavailable unless a host deliberately provides the capability. -- If `fetch` is added, model it as a host-provided outbound capability with policy controls: allowed origins, methods, headers, response size, timeout, and whether response bodies may be returned, emitted, or only summarized through a tool. +- `interpreter-support.md` is a checklist, not a changelog: check off newly added support and add unchecked items for significant gaps. Leave it alone for fixes and refactors. +- Program values are `Value` (`src/interpreter/objects.ts`). Host values are `unknown` and enter only through `fromHost` and `fromJson`; do not widen program-facing signatures to `unknown`. +- Each built-in object kind is one `Obj` subclass (`Wrapper` or `Opaque`) that overrides the hooks it needs. Ask the object instead of adding `instanceof` ladders. diff --git a/packages/codemode/interpreter-support.md b/packages/codemode/interpreter-support.md index 95fa44f663e0..db8cba20c6ae 100644 --- a/packages/codemode/interpreter-support.md +++ b/packages/codemode/interpreter-support.md @@ -8,8 +8,8 @@ standard-library surface that programs can use today, plus concrete gaps that ma - Checked items do not promise complete ECMAScript edge-case parity; known differences are stated explicitly. - Intentional boundaries are not listed as compatibility work. -When behavior changes, update this file and the tests in the same change. The implementation and tests remain the -ultimate source of truth. Upstream test262 files run verbatim from `test/test262`; a failing file is listed in +Check off items as support lands and add unchecked items for significant gaps. The implementation and tests remain +the ultimate source of truth. Upstream test262 files run verbatim from `test/test262`; a failing file is listed in `test/test262/skipped.txt` and its gap is an unchecked item here (see `test/test262/README.md`). ## Source and execution model diff --git a/packages/core/src/tool/AGENTS.md b/packages/core/src/tool/AGENTS.md index 4af998f8cb66..0a4dbd1bf81a 100644 --- a/packages/core/src/tool/AGENTS.md +++ b/packages/core/src/tool/AGENTS.md @@ -58,7 +58,3 @@ Tool filtering is catalog visibility, not execution authorization. A call still Built-ins return complete tool responses. `Tool.Snapshot.execute` is the local execution boundary. Generic output bounding is applied by the Session runner after execution. Producer capture remains local to producers. Shell stores combined process output in its backing file and returns a bounded tail with the full-output path when truncated. - -## Current Gaps - -- Future Session-scoped registrations still need an explicit canonical registration design. diff --git a/packages/schema/AGENTS.md b/packages/schema/AGENTS.md index 3b4c6a818af9..b2b12e89c400 100644 --- a/packages/schema/AGENTS.md +++ b/packages/schema/AGENTS.md @@ -21,8 +21,8 @@ ## Events - Classify event definitions by protocol role before adding them to a public manifest: `current`, `shared transitional`, or `V1-only`. -- Being emitted by V1 is not enough to include an event in Protocol or SDK Next. -- Keep clearly V1-only events, such as `message.updated` and `message.part.*`, out of the current Protocol/SDK Next event surface unless a current-client requirement is documented. +- Being emitted by V1 is not enough to include an event in Protocol or the SDK. +- Keep clearly V1-only events, such as `message.updated` and `message.part.*`, out of the current Protocol/SDK event surface unless a current-client requirement is documented. - Keep compatibility events available only to the existing App/TUI/CLI compatibility surface while they are still needed. - Preserve a single canonical event definition. Do not duplicate definitions for generation convenience.