Pouchy Companion Protocol (v1 draft)

The interface a third-party website / app / game / hardware uses to embed the Pouchy companion — chat, play, watch, spend, execute — with shared memory and bidirectional, real-time data (the app streams live context in; the companion streams actions out). Standards-first: a partner that already speaks MCP / AG-UI / CloudEvents / WebRTC / OAuth integrates with almost no Pouchy-specific learning.

Status: live in REST/SSE form. The auth layer (Personal Access Tokens + "Login with Pouchy" OAuth 2.1), session lifecycle, world-state ingest, the server agent loop that fills the outbound stream, and mid-call voice context injection are all shipped (/api/companion/session…). The WebSocket plane is not served by pouchy.ai (a serverless runtime cannot hold a socket): the client accepts stream: 'websocket' and falls back to SSE automatically against it. It IS served by an optional self-hosted WebSocket gateway worker (a small always-on process that ships with the platform source and proxies a session's SSE stream over one socket) — a host that can hold sockets runs it and points the SDK's stream at it; every frame still comes from the platform's SSE route with the same cursor, so read every "WebSocket" mention below as the same envelope/event contract SSE carries, over one socket.

0. Three planes

The control/data plane is also the signaling + bridge that feeds live world-state into the voice session (§5). That plane is REST + SSE; the WebSocket gateway (§7) carries the identical envelope over one socket for a host that runs it.

1. Authentication

Every connection carries a Personal Access Token the user generated via POST /api/companion/keys (signed-in first-party user; no in-app Wallet panel for this yet):

Authorization: Bearer pchy_<opaque-secret>

• The token represents one user; the same token = the same companion + the same memory on every surface. (Multi-user products use "Login with Pouchy" OAuth at P1 — same bearer shape downstream.)
• The token carries scopes and modalities chosen at creation. Sensitive scopes (marked below) are off unless the user opted in. The full vocabulary (source of truth: src/lib/types/companion-scopes.ts, also exported typed from the SDK as COMPANION_SCOPES / SENSITIVE_SCOPES / DEFAULT_SCOPES / hasScope(granted, required) since @pouchy_ai/companion-sdk 0.19.0, drift-guarded against the server list):
  • chat — text turns
  • voice — TTS / STT
  • call — realtime WebRTC voice session
  • files — multimodal file in/out (vision / image)
  • events.subscribe — receive the outbound companion event stream
  • worldstate.write — push inbound live world-state / context events
  • ui.render — let the companion render Instant UI surfaces on the host
  • data.activity — receive metadata-only Data-plane activity frames (capability + outcome + timing; never rows or intent values). Granted automatically from the agent template's Data flag.
  • chat.instructions — carry developer-authored per-turn instructions on a text turn (input.text / POST …/input field instructions). NOT granted by default, and the reason is the security of the field rather than caution: instructions is joined to the prompt WITHOUT being scored by inbound moderation, and the session token that reaches /input is routinely held by the end user's own client. Honouring it therefore has to be a deliberate act by whoever mints the key. How to obtain it: set instructions: true on the AGENT TEMPLATE (Dashboard → Agents → the agent, or the template PATCH) — the same genui → ui.render shape. Asking for it in the mint body does nothing: validateSessionRequest narrows requested scopes to the default set, which this one is deliberately outside of.
  • memory.read:app / memory.write:app — this app's own memory namespace
  • memory.read:core / memory.write:core — the shared global brain (SENSITIVE)
  • skills.execute — run the user's installed skills (SENSITIVE)
  • wallet.read — read-only balance + own deposit address (receive-only)
  • wallet.spend — spend from the Care Wallet (SENSITIVE)
  • social.message — message friends via A2A (SENSITIVE)
  • represent — open visitor-facing representative sessions (SENSITIVE)
  • expose:knowledge — let the representative draw on the owner's knowledge base (SENSITIVE)
  • expose:facts — widen exposed facts past the identity/preference floor (SENSITIVE)
  • represent:pair — let a representative pair a visitor as a friend (SENSITIVE)
  • represent:remember — durable per-visitor notes across visits (SENSITIVE)
• WS upgrade and every REST call verify the token via requireAppToken (src/lib/server/app-token.ts). 401 = missing/invalid/revoked; 403 = missing a required scope.

2. Message envelope (control/data plane)

All control/data-plane messages (SSE today, WebSocket when it lands) share one JSON envelope (AG-UI-aligned event types):

{
  "v": 1,                       // protocol version
  "id": "msg_01H…",             // sender-generated; idempotency key for effectful inbound
  "session": "sess_01H…",       // assigned by hello.ack
  "ts": 1718600000123,          // unix ms
  "type": "context.event",      // see §3
  "payload": { /* per-type */ }
}

Conventions: text/JSON for web/app; protobuf for the same schema on bandwidth- constrained hardware (§7). Unknown types are ignored (forward-compat).

3. Message types

Inbound (app → Pouchy)

Outbound (Pouchy → app)

control.error codes (a separate vocabulary from the HTTP {ok:false, code} tags in §9 — these arrive ON the stream, surfaced by the SDK's onError):

This table is exported typed since SDK 0.30.0: CONTROL_ERROR_CODES (the server-emitted rows; runtime source of truth is src/lib/types/companion-protocol.ts, mirrored into the SDK and drift-tested — including a scan of every emit site's code literal) plus the ControlErrorCodeValue union onError's err.code carries (server codes + the SDK-synthesized stream_unauthorized + an open arm for codes newer than the installed SDK build). The vocabulary is APPEND-ONLY.

companion.tool_call is offered only for tools the app declared in the hello/session tools[] ∩ the token's scopes — the agent can't ask a game to run a payment it didn't opt into. (handles[] is stored but never consumed server-side today — treat it as a client-side capability hint; a tool the app wants called MUST be declared in tools[].)

The name you get back is always the name you declared, byte for byte, so onToolCall can switch on it directly. Providers cap a function name at 64 characters and the session handshake truncates there, but the truncation is a provider-side detail: the relayed companion.tool_call, the /input response and the hello.ack.pendingToolCalls replay all speak your name. (Two declared names that differ only past character 64 still collapse to one — the provider rejects a duplicate function name outright — so keep the first 64 distinct.)

3.1 Token streaming (the input POST response)

The reply can render as it is generated instead of arriving as one block. Deltas do NOT travel on the outbound event log (that would mean a durable write per chunk plus poll latency) — they ride the very request that runs the turn: POST …/input with { stream: true } answers with text/event-stream:

The success done frame also carries first-party diagnostic fields — timings (server-side phase breakdown incl. server_first_delta_ms / server_total_ms) and promptCache (the final hop's prefix-cache health) — and the failure shape carries errorId (the sanitized server-error reference). They are additive telemetry for Pouchy's own perf probes, not contract: receivers ignore unknown fields by doctrine, and third-party clients should not build on their shape.

The success done frame additionally carries a usage block for per-turn cost accounting — { promptTokens, completionTokens, cachedTokens } (whole-turn sums across every model hop, so they equal the dashboard trace's tokensIn / tokensOut), plus reasoningTokens (a subset of completionTokens, not additive — present only when the served model reports it) and model (the served model, fallback-aware — present when known). The same usage block is returned on the non-stream JSON ack ({ ok, kind: 'message', seq, usage }) so a host that does not open the SSE stream still gets it. It is omitted / null on the failure and duplicate acks. Unlike promptCache, this block is a stable additive field integrators may persist; it stays out of the versioned envelope union (PROTOCOL_VERSION unchanged) because it is a transport feature of the input endpoint, not a wire event.

Contract: deltas are advisory; the final companion.message is authoritative (the server strips its internal state block from the final text only). The done frame carries the full envelope so a client can emit it locally (zero poll latency) and drop the event-log replay by envelope id — the SDK's onDelta + sendText do exactly this, and onMessage still fires exactly once per turn. Errors after the stream opens arrive in the done frame (the HTTP status line is already sent). No envelope-union / PROTOCOL_VERSION change — this is a transport feature of the input endpoint.

4. World-state stream — the play-along channel (the crux)

This is what makes 陪玩/陪看 good: the app streams what's happening; the companion reacts. Use a CloudEvents-shaped envelope inside context.event.payload:

{
  "specversion": "1.0",
  "id": "evt_01H…",
  "source": "game://my-rpg/session/42",
  "type": "game.event.boss_spawned",      // namespaced kind (see below)
  "time": "2026-06-17T12:00:00Z",
  "data": { "boss": "Dragon", "arena": "north" },

  // Pouchy extensions:
  "retained": false,        // true = latest-value state; false = transient moment
  "ttl": 5000,              // ms the value stays relevant
  "salience": 0.9,          // 0..1 — how much this should grab the companion
  "voiceRelevant": true     // push into a live voice call (§5)
}

Kind taxonomy (namespaced, dotted): game.player.hp, game.player.location, game.event.boss_spawned, game.event.level_up, media.playback.position, media.scene.changed, shop.cart.updated, … Apps declare the kinds they emit in hello.contextKinds.

Two flavors:

• Retained state (retained:true): current values (hp, scene, location). The server keeps the latest per kind in a short-TTL per-session buffer; coalesced so a value changing every frame doesn't wake the agent.
• Transient events (retained:false): discrete moments. Carry salience and, if it should interrupt a conversation, voiceRelevant.

Ambient vs. trigger: a kind is either ambient (consulted at the next turn — "knows your hp when you ask") or a trigger (proactively prompts a companion reaction — "boss incoming, heal!"). Declared per kind at app registration.

Server handling: raw telemetry never enters the prompt. The server maintains a rolling compact digest of world-state and injects only that (arbitrated by the prompt's context budget). salience + per-token rate limits throttle the firehose and protect the bill. Authenticated + replay-deduped with the same HMAC machinery as webhooks (src/lib/server/webhook-verify.ts).

5. Realtime voice ↔ world-state (call mode)

In call mode the audio is WebRTC peer-to-peer to the voice provider (ElevenLabs Convai, or OpenAI Realtime as fallback) — Pouchy mints a short-lived token and steps out of the media path:

app → control.start_call
Pouchy → control.call_ready { provider: "elevenlabs-convai", agentId: "…", … }   (secret-free; creds ride the HTTP response)
app → opens WebRTC to the provider with that token

The hard part — and the highest lever on quality — is feeding live game context into the in-call model even though Pouchy isn't in the audio loop:

• When a context.event arrives with voiceRelevant: true, the server pushes a compact contextual update into the live voice session over the provider's data channel — ElevenLabs contextual updates, or OpenAI Realtime conversation.item.create + response.create. So the companion can say "Boss incoming — heal up!" mid-call. This routing is gated on a call-active window the /call mint opens (≤20 min) and POST /end closes — connectCall().close() sends that automatically via endSession(), and since SDK 0.40.1 a connect-phase failure after the mint tears it down the same way; self-plumbed startCall integrations should call endSession() at hang-up (and on a failed connect) so text play-along reactions resume immediately instead of after the window lapses. The clear is generation-fenced (SDK 0.41.0): /call returns callGen, /end echoes it — a mismatch (stale teardown after a re-mint) no-ops, an explicit callGen: null (a text client's teardown) leaves a live window alone, and an omitted field keeps the pre-0.41 unconditional clear.
• Updates are debounced/coalesced so we never spam the voice model.
• Today the persona/memory prompt is injected only at connect (src/lib/services/convai/connection.ts, …/realtime/connection.ts); mid-call injection is net-new and the key thing to validate first.
• Fallback for tight coupling (rhythm games, etc.): route audio through a server agent (LiveKit) where Pouchy fully controls the loop — at latency/cost expense. P3 only.

6. Standards mapping

Bespoke (no standard — carried inside the above): the world-state kind taxonomy, the per-app memory scope/consent model, persona/relationship-stage.

7. Hardware / constrained clients

Same envelope and types, but:

• protobuf encoding option for the WS frames.
• SSE + REST long-poll fallback when WS/WebRTC aren't available.
• MCP over Streamable HTTP as the agent-interop variant (a device that is itself an agent).
• Server-driven TTS/STT (/api/tts, /api/stt) when the device can't run a WebRTC voice session. Both are auth-gated and metered: send a companion token carrying the voice scope (it covers TTS and STT — a token granted only chat/memory must not drive operator-billed transcription).

8. Cross-cutting

• Handshake (hello → hello.ack) negotiates modalities, declared action handlers, context kinds, granted scopes, and the protocol version.

• Resume: hello.ack.resumeCursor; on reconnect the client replays missed outbound events by cursor. Heartbeats via control.ping.

  The cursor is a sequence number, not a time. On the SSE transport the server writes it on each event frame's id: line and repeats the window's final value in the closing reconnect frame's { cursor }; a client resumes with ?cursor=<n> (or Last-Event-ID), and the server answers with the events whose seq is strictly greater. Take it from those two places only — in particular never from an envelope's ts, which is a wall-clock Date.now() stamp: feeding one back as a cursor asks for a sequence past 1.7e12 and matches nothing, so the stream goes silent from the first reconnect onward with no error anywhere. (The Go SDK did exactly this through 0.1.1; see its 0.1.2 entry.)

• Client-restart recovery of a paused turn (SDK 0.37.0): a turn paused on the app's declared tools survives the app itself dying. Discovery: hello.ack.pendingToolCalls lists the still-outstanding calls when the handshake resumes a session mid-pause. Completion: perform each call and post tool.result per id (idempotent — a double post is safe), and the turn resumes normally. The SDK automates this: unless replayPendingToolCalls: false, each pending call is re-emitted to the app's onToolCall handler with replayed: true one tick after connect() resolves (at most once per client instance), so the existing tool loop completes the turn. Abandon path (unchanged, SDK 0.36-era): /end clears the paused turn and unblocks the session's 409 turn_pending — recovery now makes that the fallback, not the only option.

• Idempotency: effectful inbound (tool.result, effectful context.event) carries the envelope id; the server dedupes so SDK retries don't double-apply. For /context the dedup window is 60 min and the response's duplicates field reports how much of accepted was replay dedup — stamp STABLE ids on host-level retries (an omitted id is minted fresh per POST and never dedups).

• Conversation continuity: the server keeps a rolling per-session turn log — each text turn reasons over the last ~12 exchanges (including approved confirm outcomes), and end-of-session memory consolidation falls back to this log when the client sends no transcript. Embedders do NOT need to resend history with each message.

• Sensitive actions (spend / message a friend / run a skill) require a confirm, enforced server-side. WHO approves depends on the token: a first-party user's ops confirm only on a Pouchy-controlled surface (hosted widget or their app/phone — never trust the third-party DOM); a platform instance's ops (session tokens minted via /v1/sessions) confirm via the session itself (client.confirmAction / POST /api/companion/session/{id}/confirm) — the embedding app's end user is that account's only human, and instances are never granted wallet.spend, so this path can't move money. Pending list for rebuilds: client.pendingConfirms() (GET on the same endpoint).

• Errors: typed control.error { code, message }; HTTP uses standard status + { ok: false, error, code? } JSON — code (v0.21+) is a stable machine-readable tag (invalid_token, missing_scope, session_not_found, turn_pending, …; full table in companion-api-versioning.md) surfaced by the SDK as CompanionError.code. The SDK also synthesizes client-side codes outside that HTTP vocabulary: reply_timeout (awaitReply fallback), needs_event_stream (awaitReply on a tool-pausing turn without start(), SDK 0.28.0), not_connected / missing_option (client-side misuse guards), not_representative (pairVisitor on a non-representative session), aborted (a caller-supplied AbortSignal cancelled the request, SDK 0.29.0), queued_offline (queued while offline, SDK 0.34.0), request_timeout (the requestTimeoutMs headers deadline elapsed, SDK 0.35.0; since SDK 0.40.0/0.40.1 the long-work POSTs default to their route's own ceiling — buffered /input sends, sendToolResult and the knowledge ingests 310 s, confirmAction 65 s — their headers only arrive when the server work completes), stream_unauthorized (stream 401 exhausted reconnects), and the voice connect-step codes (SDK 0.26.0) call_unsupported / call_connect_failed / call_dependency_missing on connectCall/startCall rejections. From SDK 0.28.1 the full list is drift-tested (SDK_SYNTHESIZED_ERROR_CODES).

9. End-to-end example — a game that plays along

1. app → hello { modalities:{text,call},
                 tools:[{ name:"game.highlight", description:"highlight a UI element",
                          parameters:{ type:"object", properties:{ target:{ type:"string" } } } }],
                 contextKinds:["game.player","game.event.*"] }
   (`tools[]` is what makes companion.tool_call possible — `handles[]` alone
    does not register a tool server-side)
   ←  hello.ack { session, grantedScopes, modalities, resumeCursor }

2. app → control.start_call
   ←  control.call_ready { provider:"elevenlabs-convai", agentId:"…" }  (secret-free)
   app opens WebRTC to ElevenLabs.

3. during play:
   app → context.event { type:"game.player.hp", retained:true, ttl:5000,
                         data:{ hp:80 } }                 // coalesced, ambient
   app → context.event { type:"game.event.boss_spawned", salience:0.9,
                         voiceRelevant:true, data:{ boss:"Dragon" } }

4. server pushes a contextual update into the live voice session →
   companion SAYS: "Boss incoming — heal up first!"

5. companion → companion.tool_call { id:"tc_1", name:"game.highlight",
                                     args:"{\"target\":\"healPotion\"}" }
   (`args` is the model's JSON **string** — parse it, or use the SDK's
   pre-parsed `argsJson`)
   app highlights the potion, then:
   app → tool.result { callId:"tc_1", ok:true }

The tighter steps 3-4 are wired (salience, coalescing, voiceRelevant → injection), the better the companionship feels — which is exactly why this plane gets the most care.