Companion Instant UI — the cross-platform renderer contract

Instant UI lets the companion draw a small interactive panel during a turn instead of (or alongside) speaking text — a form to collect a few values, a summary card with a progress bar, a chart, a set of choices. Over the SDK this arrives as a companion.ui_action event, and your host renders it.

The whole point: the payload is platform-neutral JSON. The exact same interface is drawn by a web embed, a native iOS / Android view tree, or a CLI / TUI — only the renderer differs per platform, never the contract. There is no HTML, no Svelte, no DOM in the wire format; it is a tree of typed UI atoms plus a state bag.

In the first-party Pouchy app the same schema is drawn by a Svelte renderer (src/lib/components/genui/DynamicRenderer.svelte). A native/CLI host writes the equivalent ~20-atom renderer once and gets every panel the companion will ever emit.

1. Enable it

Mint the session token with the ui.render scope (non-sensitive; opt-in). An app that owns its own UI and doesn't want the companion drawing panels simply omits it. To receive the event stream you also need events.subscribe.

const c = createCompanion({ baseUrl, token }); // token granted ui.render
await c.connect();
c.onRender(({ interface: ui }) => myRenderer.draw(ui)); // your renderer
c.start();
await c.sendText('help me split the bill 4 ways');

onRender is sugar over on('companion.ui_action', …) with a guard that drops malformed frames. The payload is RenderInterfacePayload.

2. The payload

{
  "interface": {
    "title": "Split the bill",          // optional panel title
    "nodes": [ /* the UI atom tree */ ],  // 1..20 atoms (required)
    "state": { "people": 4, "total": 0 },// initial values for bound inputs / templates
    "computed": {                          // derived, read-only values
      "each": { "op": "div", "args": ["total", "people"], "round": 2 }
    },
    "reportChanges": true                  // stream the user's edits back as a turn
  }
}

• state is the bag input atoms read and write, and that {key} templates reference. Values are string | number | boolean.
• computed are derived values recomputed from state on every change — op is mul | add | sub | div, args are state keys or literal numbers, round fixes display decimals. Reference a computed key exactly like a state key. (No expression language — keep it to these four ops.)
• reportChanges — see §5 (the live-form loop).

The payload is already validated and clamped server-side (node budgets, src allow-lists, safe markdown, dropped unknown atoms). A renderer can trust the shape and never needs to sanitize model output itself.

3. The atoms

Each node is { type, …props }. Any node may also carry a when clause (§4). Three buckets:

Display (read-only)

Input (read + write a state key)

Layout (contain children)

A renderer that implements these renders every panel the companion can emit. If your platform genuinely can't draw one (e.g. no inline video on a TUI), render a graceful placeholder — the rest of the tree is unaffected.

4. Conditional nodes (when)

Any node may carry when: { key, equals?, truthy? } and is shown only while the clause holds against live state. equals wins if present (value match); else truthy gates on Boolean(state[key]) (default true). This lets a Switch or Select reveal/hide a section reactively without the model rebuilding the panel — re-evaluate when on every state change.

5. The live-form loop (reportChanges)

When interface.reportChanges is true, the host should send the user's edited state back to the companion as the next turn so it can react to the adjustments — "the AI follows your slider". The natural transport is a normal text turn that states the chosen values, or a context.snapshot; the model then continues with the real numbers in hand. When reportChanges is absent/false, the panel is local — edits stay on the host until an explicit submit (a Button, §6).

This is what makes a form work without any button-action dispatch: collect, report, the model replies. It is the recommended interaction pattern over the SDK.

5b. Live updates (companion.ui_update)

After rendering a panel, the companion can change it in place instead of re-rendering: an update_interface tool call emits a companion.ui_update event carrying { update: { panelId?, updates: [{ key, value }] } }. Apply each write to the rendered panel's state bag and re-evaluate the bound displays ({key} templates, Progress, when visibility) — inputs keep their focus, nothing rebuilds. panelId targets one of several on-screen panels when present.

let panel: PanelHandle | null = null;
c.onRender(({ interface: ui }) => { panel = renderInterface(ui, mount); });
c.onInterfaceUpdate(({ update }) => panel?.applyUpdate(update));

On native, the reactive state already drives this: write the keys into the @Observable (SwiftUI) / snapshot-state (Compose) bag and the views recompose.

6. Buttons & actions (advanced)

A Button.action is an AgentAction — a request to run a registered action (after { $state: key } params are resolved to current values). Over the SDK an action whose type matches a tool the app declared surfaces as a companion.tool_call you already handle; a confirm-gated action still runs its confirmation. For v1, prefer the reportChanges loop (§5) for collecting input; reach for action buttons only when a tap must trigger a declared app tool.

7. Rendering on each platform

• Web — a component per atom, a reactive store for state, recompute computed + re-evaluate when on change. (Reference: the first-party Svelte renderer.)
• Native (iOS / Android) — a view per atom (SwiftUI View / Jetpack Compose @Composable), a state view-model, two-way bindings for the input atoms. The atom list maps cleanly to native controls (Slider→Slider, Switch→Toggle/Switch, Select→Picker/menu, DatePicker→native date UI).
• CLI / TUI — render display atoms as text/boxes; drive input atoms with prompts (a Select becomes a numbered menu, a Slider a number prompt). Layout atoms become indentation/sections. A headless agent can even render to plain text and feed answers back via reportChanges.

Standard wire format option

If you'd rather not bind to Pouchy's atom names, the schema projects losslessly (modulo a few _pouchy extension keys) to Google's A2UI v0.9 flat wire format via src/lib/genui/a2ui-adapter.ts (toA2UI / fromA2UI). A renderer built against the A2UI catalog can consume Pouchy Instant UI with that adapter in front.

8. Why this is "beyond the sandbox"

The first-party app renders Instant UI inside the browser. The SDK is not bound to that: because the contract is pure data, the companion's generative UI runs wherever the host can draw ~20 atoms — a native app, a game engine's UI layer, a terminal, a kiosk. Same companion, same panels, any surface. The renderer is the only platform-specific part, and it's written once.