Companion SDK — the six capabilities, and how each goes beyond the sandbox

The companion exposes six capabilities. Each is reachable over the SDK, and each has a platform-neutral contract — so the first-party web app is just one host. A native iOS/Android app, a CLI, a game engine, or a kiosk can use the same capability; only the renderer/executor differs per platform.

Capability SDK surface Scope Beyond the web sandbox
Memory (记忆) remember tool · recall_memory tool (the model's own on-demand search — every turn already carries a ranked window, this is for what is older or more specific than that) · recall() memory.read/write:app, :core Shared core brain recalled across every app, not one browser's IndexedDB
Reasoning (推理) the server agent loop (every turn) · generate_image (agent-opt-in imageGen; server-side generation, replies carry a 7-day capability URL) · find_apps (app-directory lookup — surfaces a launch URL for what the user wants to do) · create_spreadsheet / create_document (files scope; real .xlsx/.csv/.docx/.md/.txt output — 7-day capability URL in the reply, structural attachments[] on the envelope since SDK 0.54.0, and a native IM attachment on channels that support one) chat (+files) Same model loop drives any host; tools can reach native capabilities
Skills (技能) get_skills (list; { skill, doc } fetches one bundled reference doc) · learn_skill (install from an https skill.md / zip bundle / ClawHub URL — guarded fetch + safety gate, rate-limited, reversible) · forget_skill · run_skill · http_request (free-HTTP — drive an armed skill's API from its prose body; GET/HEAD auto-run, mutations confirm-gated; see skills.md §6b) · read_skill_resource · get_skill_prompt; host-declared tools; confirmAction / pendingConfirms (platform sessions resolve confirm-gated runs) skills.execute HTTP/MCP and local/device commands a host declares (shell, clipboard, notifications, camera…); platform custom skills split into auto-run (credential-free GET) vs confirm-gated (POST / vault-credentialed); a docs-only skill runs via free-HTTP once its owner arms it
A2A / Social (社交) get_friends · send_friend_message · message_friends · read_friend_messages; onSocialMessage social.message Friend messages delivered cross-app over the event stream, not just the first-party Social view
Wallet (钱包) get_wallet_balance · get_deposit_address · pay_friend · pay_address; onConfirmRequest wallet.read (balance + deposit address, read-only) / wallet.spend (payments) Biometric confirm on a native surface for spends; wallet.read alone can never move funds (platform instances get read-only)
Instant UI (即时UI) render_interface tool → companion.ui_action · update_interface (in-place state-bag writes to an already-rendered panel); onRender ui.render One {interface} JSON renders on web, iOS, Android, and CLI; toA2UI/fromA2UI (SDK 0.42.0) project it to/from Google's A2UI v0.9 wire format for external A2UI-catalog renderers

Agent Data — the platform data plane (View / Action / Event)

Not a seventh capability: Agent Data is platform infrastructure underneath the six — the controlled bridge between an agent and the developer's own authoritative backend. It groups three primitives that are agent-world semantics, not HTTP methods:

  • View — read authoritative state (read_data; results curated to declared fields, identity minted server-side and signed).
  • Action — change external state safely (run_action; user-confirmed, idempotent by a runtime-minted actionId, honest unknown outcomes).
  • Event — the external world tells the agent something changed (POST /v1/projects/{id}/events; a deduped signal, never a payload transport — current facts still come from Views).

Capabilities are declared per project (dashboard → Data Capabilities, or POST /v1/projects/{id}/capabilities; published revisions are immutable) and granted per agent by the data flags (enabled is the parent gate; actions / events are second privileges under it). Hands-on walkthrough: Quickstart: Agent Data Plane; contract details: API Reference (Data capabilities section).

How a capability "transcends" — the pattern

Every capability is data + a renderer/executor:

  • The server produces platform-neutral output (a memory fact, a tool-call request, an {interface} tree, a friend message) and enforces the safeguards (scopes, spend caps, confirm-gate, SSRF/domain allowlists).
  • The host consumes it however its platform allows. The browser is one host; it has no special status.

So "Instant UI on iOS", "skills as CLI commands", "cross-app A2A" aren't new engines — they're new hosts of contracts that were already platform-neutral.

Events an embed receives (companion.*)

hello.ack · companion.message · companion.audio (a TTS clip of the reply, opt-in per agent) · companion.tool_call · companion.ui_action · companion.ui_update · companion.social_message · companion.confirm_request · companion.expression (an avatar cue, opt-in per agent) · companion.voice_inject · companion.typing · companion.data_activity (metadata-only Data-plane activity; requires the data.activity scope) · control.*. Subscribe with on(type, fn) or the typed helpers (onMessage / onToolCall / onRender / onSocialMessage / onConfirmRequest / onTyping / onVoiceInject). See companion-sdk-protocol.md.

Live voice. connectCall() opens a realtime call; since SDK 0.64.0 connectCall({ audioTags: true }) lets the voice perform Eleven v4 cues — it laughs, sighs and whispers instead of only speaking words — with the cues stripped from onTranscript and from the memory transcript for you (stripAudioTags for self-plumbed sessions). Since SDK 0.70.0 each companion line on a granted call also carries onTranscript's audioTags: string[] — the cues it performed, in order ([] when none); a self-plumbed session gets the same list from the exported extractAudioTags(text), which matches exactly what stripAudioTags removes, so do not write your own cue regex. Opt-in; ElevenLabs calls only. Since SDK 0.65.0 connectCall({ record: true }) records the whole call, both voices, on the provider side — so a native app that cannot capture call audio gets the same file a browser could — and getCallRecording(callGen) returns the MP3 after the call (kept 7 days; deleteCallRecording removes it). Turned on per agent in the dashboard (the agent's recording switch). Since SDK 0.75.0 connectCall({ avatar: { faceId } }) adds a talking-face video avatar (Simli) to the call: call.video carries the session you open with simli-client, fed the companion's remote audio track. You build the faces yourself with the project's admin key (admin-sdk createVideoFace, billed per face built); the call.video scope can be withheld per reader to keep the voice and drop the face. Billed per minute on top of voice. avatar.model (0.76.0) picks Simli's animation model, fasttalk (default) or artalk. Since SDK 0.67.0 connectCall({ brief }) tells the companion what a call is about (your application's note; needs chat.instructions), and a call made by a Pouchy World role carries where the story is, as that role sees it, from the server (call.worldBrief). The call does not write the world; fold it back in as a beat with trigger.context + proposedPatches.

Reference renderers & hosts

Surface Where
Instant UI — web (framework-free) examples/web-instant-ui
Instant UI — iOS (SwiftUI) examples/native-instant-ui/InstantUI.swift
Instant UI — Android (Compose) examples/native-instant-ui/InstantUI.kt
Skills + Instant UI — CLI/terminal examples/cli-skills
Voice play-along — web examples/poker-voice

Deep dives