Pouchy Companion Protocol (v1 draft)

The interface a third-party website / app / game / hardware uses to embed the Pouchy companion — chat, play, watch, spend, execute — with shared memory and bidirectional, real-time data (the app streams live context in; the companion streams actions out). Standards-first: a partner that already speaks MCP / AG-UI / CloudEvents / WebRTC / OAuth integrates with almost no Pouchy-specific learning.

Status: live in REST/SSE form. The auth layer (Personal Access Tokens + "Login with Pouchy" OAuth 2.1), session lifecycle, world-state ingest, the server agent loop that fills the outbound stream, and mid-call voice context injection are all shipped (/api/companion/session…). The WebSocket plane is not yet served: the client accepts stream: 'websocket' and falls back to SSE automatically, but the server exposes no WS endpoint today — read every "WebSocket" mention below as the planned upgrade of the same envelope/event contract, which SSE carries unchanged in the meantime.

0. Three planes

The control/data plane is also the signaling + bridge that feeds live world-state into the voice session (§5). Today that plane is REST + SSE; the WebSocket upgrade will carry the identical envelope when it lands (§7).

1. Authentication

Every connection carries a Personal Access Token the user generated via POST /api/companion/keys (signed-in first-party user; no in-app Wallet panel for this yet):

Authorization: Bearer pchy_<opaque-secret>

• The token represents one user; the same token = the same companion + the same memory on every surface. (Multi-user products use "Login with Pouchy" OAuth at P1 — same bearer shape downstream.)
• The token carries scopes and modalities chosen at creation. Sensitive scopes (marked below) are off unless the user opted in. The full vocabulary (source of truth: src/lib/types/companion-scopes.ts, also exported typed from the SDK as COMPANION_SCOPES / SENSITIVE_SCOPES / DEFAULT_SCOPES / hasScope(granted, required) since @pouchy_ai/companion-sdk 0.19.0, drift-guarded against the server list):
  • chat — text turns
  • voice — TTS / STT
  • call — realtime WebRTC voice session
  • files — multimodal file in/out (vision / image)
  • events.subscribe — receive the outbound companion event stream
  • worldstate.write — push inbound live world-state / context events
  • ui.render — let the companion render Instant UI surfaces on the host
  • memory.read:app / memory.write:app — this app's own memory namespace
  • memory.read:core / memory.write:core — the shared global brain (SENSITIVE)
  • skills.execute — run the user's installed skills (SENSITIVE)
  • wallet.read — read-only balance + own deposit address (receive-only)
  • wallet.spend — spend from the Care Wallet (SENSITIVE)
  • social.message — message friends via A2A (SENSITIVE)
  • represent — open visitor-facing representative sessions (SENSITIVE)
  • expose:knowledge — let the representative draw on the owner's knowledge base (SENSITIVE)
  • expose:facts — widen exposed facts past the identity/preference floor (SENSITIVE)
  • represent:pair — let a representative pair a visitor as a friend (SENSITIVE)
  • represent:remember — durable per-visitor notes across visits (SENSITIVE)
• WS upgrade and every REST call verify the token via requireAppToken (src/lib/server/app-token.ts). 401 = missing/invalid/revoked; 403 = missing a required scope.

2. Message envelope (control/data plane)

All control/data-plane messages (SSE today, WebSocket when it lands) share one JSON envelope (AG-UI-aligned event types):

{
  "v": 1,                       // protocol version
  "id": "msg_01H…",             // sender-generated; idempotency key for effectful inbound
  "session": "sess_01H…",       // assigned by hello.ack
  "ts": 1718600000123,          // unix ms
  "type": "context.event",      // see §3
  "payload": { /* per-type */ }
}

Conventions: text/JSON for web/app; protobuf for the same schema on bandwidth- constrained hardware (§7). Unknown types are ignored (forward-compat).

3. Message types

Inbound (app → Pouchy)

Outbound (Pouchy → app)

control.error codes (a separate vocabulary from the HTTP {ok:false, code} tags in §9 — these arrive ON the stream, surfaced by the SDK's onError):

This table is exported typed since SDK 0.30.0: CONTROL_ERROR_CODES (the server-emitted rows; runtime source of truth is src/lib/types/companion-protocol.ts, mirrored into the SDK and drift-tested — including a scan of every emit site's code literal) plus the ControlErrorCodeValue union onError's err.code carries (server codes + the SDK-synthesized stream_unauthorized + an open arm for codes newer than the installed SDK build). The vocabulary is APPEND-ONLY.

companion.tool_call is offered only for tools the app declared in the hello/session tools[] ∩ the token's scopes — the agent can't ask a game to run a payment it didn't opt into. (handles[] is stored but never consumed server-side today — treat it as a client-side capability hint; a tool the app wants called MUST be declared in tools[].)

3.1 Token streaming (the input POST response)

The reply can render as it is generated instead of arriving as one block. Deltas do NOT travel on the outbound event log (that would mean a durable write per chunk plus poll latency) — they ride the very request that runs the turn: POST …/input with { stream: true } answers with text/event-stream:

The success done frame also carries first-party diagnostic fields — timings (server-side phase breakdown incl. server_first_delta_ms / server_total_ms) and promptCache (the final hop's prefix-cache health) — and the failure shape carries errorId (the sanitized server-error reference). They are additive telemetry for Pouchy's own perf probes, not contract: receivers ignore unknown fields by doctrine, and third-party clients should not build on their shape.

Contract: deltas are advisory; the final companion.message is authoritative (the server strips its internal state block from the final text only). The done frame carries the full envelope so a client can emit it locally (zero poll latency) and drop the event-log replay by envelope id — the SDK's onDelta + sendText do exactly this, and onMessage still fires exactly once per turn. Errors after the stream opens arrive in the done frame (the HTTP status line is already sent). No envelope-union / PROTOCOL_VERSION change — this is a transport feature of the input endpoint.

4. World-state stream — the play-along channel (the crux)

This is what makes 陪玩/陪看 good: the app streams what's happening; the companion reacts. Use a CloudEvents-shaped envelope inside context.event.payload:

{
  "specversion": "1.0",
  "id": "evt_01H…",
  "source": "game://my-rpg/session/42",
  "type": "game.event.boss_spawned",      // namespaced kind (see below)
  "time": "2026-06-17T12:00:00Z",
  "data": { "boss": "Dragon", "arena": "north" },

  // Pouchy extensions:
  "retained": false,        // true = latest-value state; false = transient moment
  "ttl": 5000,              // ms the value stays relevant
  "salience": 0.9,          // 0..1 — how much this should grab the companion
  "voiceRelevant": true     // push into a live voice call (§5)
}

Kind taxonomy (namespaced, dotted): game.player.hp, game.player.location, game.event.boss_spawned, game.event.level_up, media.playback.position, media.scene.changed, shop.cart.updated, … Apps declare the kinds they emit in hello.contextKinds.

Two flavors:

• Retained state (retained:true): current values (hp, scene, location). The server keeps the latest per kind in a short-TTL per-session buffer; coalesced so a value changing every frame doesn't wake the agent.
• Transient events (retained:false): discrete moments. Carry salience and, if it should interrupt a conversation, voiceRelevant.

Ambient vs. trigger: a kind is either ambient (consulted at the next turn — "knows your hp when you ask") or a trigger (proactively prompts a companion reaction — "boss incoming, heal!"). Declared per kind at app registration.

Server handling: raw telemetry never enters the prompt. The server maintains a rolling compact digest of world-state and injects only that (arbitrated by the prompt's context budget). salience + per-token rate limits throttle the firehose and protect the bill. Authenticated + replay-deduped with the same HMAC machinery as webhooks (src/lib/server/webhook-verify.ts).

5. Realtime voice ↔ world-state (call mode)

In call mode the audio is WebRTC peer-to-peer to the voice provider (ElevenLabs Convai, or OpenAI Realtime as fallback) — Pouchy mints a short-lived token and steps out of the media path:

app → control.start_call
Pouchy → control.call_ready { provider: "elevenlabs-convai", agentId: "…", … }   (secret-free; creds ride the HTTP response)
app → opens WebRTC to the provider with that token

The hard part — and the highest lever on quality — is feeding live game context into the in-call model even though Pouchy isn't in the audio loop:

• When a context.event arrives with voiceRelevant: true, the server pushes a compact contextual update into the live voice session over the provider's data channel — ElevenLabs contextual updates, or OpenAI Realtime conversation.item.create + response.create. So the companion can say "Boss incoming — heal up!" mid-call.
• Updates are debounced/coalesced so we never spam the voice model.
• Today the persona/memory prompt is injected only at connect (src/lib/services/convai/connection.ts, …/realtime/connection.ts); mid-call injection is net-new and the key thing to validate first.
• Fallback for tight coupling (rhythm games, etc.): route audio through a server agent (LiveKit) where Pouchy fully controls the loop — at latency/cost expense. P3 only.

6. Standards mapping

Bespoke (no standard — carried inside the above): the world-state kind taxonomy, the per-app memory scope/consent model, persona/relationship-stage.

7. Hardware / constrained clients

Same envelope and types, but:

• protobuf encoding option for the WS frames.
• SSE + REST long-poll fallback when WS/WebRTC aren't available.
• MCP over Streamable HTTP as the agent-interop variant (a device that is itself an agent).
• Server-driven TTS/STT (/api/tts, /api/stt) when the device can't run a WebRTC voice session. Both are auth-gated and metered: send a companion token carrying the voice scope (it covers TTS and STT — a token granted only chat/memory must not drive operator-billed transcription).

8. Cross-cutting

• Handshake (hello → hello.ack) negotiates modalities, declared action handlers, context kinds, granted scopes, and the protocol version.
• Resume: hello.ack.resumeCursor; on reconnect the client replays missed outbound events by cursor. Heartbeats via control.ping.
• Client-restart recovery of a paused turn (SDK 0.37.0): a turn paused on the app's declared tools survives the app itself dying. Discovery: hello.ack.pendingToolCalls lists the still-outstanding calls when the handshake resumes a session mid-pause. Completion: perform each call and post tool.result per id (idempotent — a double post is safe), and the turn resumes normally. The SDK automates this: unless replayPendingToolCalls: false, each pending call is re-emitted to the app's onToolCall handler with replayed: true one tick after connect() resolves (at most once per client instance), so the existing tool loop completes the turn. Abandon path (unchanged, SDK 0.36-era): /end clears the paused turn and unblocks the session's 409 turn_pending — recovery now makes that the fallback, not the only option.
• Idempotency: effectful inbound (tool.result, effectful context.event) carries the envelope id; the server dedupes so SDK retries don't double-apply.
• Conversation continuity: the server keeps a rolling per-session turn log — each text turn reasons over the last ~12 exchanges (including approved confirm outcomes), and end-of-session memory consolidation falls back to this log when the client sends no transcript. Embedders do NOT need to resend history with each message.
• Sensitive actions (spend / message a friend / run a skill) require a confirm, enforced server-side. WHO approves depends on the token: a first-party user's ops confirm only on a Pouchy-controlled surface (hosted widget or their app/phone — never trust the third-party DOM); a platform instance's ops (session tokens minted via /v1/sessions) confirm via the session itself (client.confirmAction / POST /api/companion/session/{id}/confirm) — the embedding app's end user is that account's only human, and instances are never granted wallet.spend, so this path can't move money. Pending list for rebuilds: client.pendingConfirms() (GET on the same endpoint).
• Errors: typed control.error { code, message }; HTTP uses standard status + { ok: false, error, code? } JSON — code (v0.21+) is a stable machine-readable tag (invalid_token, missing_scope, session_not_found, turn_pending, …; full table in companion-api-versioning.md) surfaced by the SDK as CompanionError.code. The SDK also synthesizes client-side codes outside that HTTP vocabulary: reply_timeout (awaitReply fallback), needs_event_stream (awaitReply on a tool-pausing turn without start(), SDK 0.28.0), not_connected / missing_option (client-side misuse guards), not_representative (pairVisitor on a non-representative session), aborted (a caller-supplied AbortSignal cancelled the request, SDK 0.29.0), queued_offline (queued while offline, SDK 0.34.0), request_timeout (the requestTimeoutMs headers deadline elapsed, SDK 0.35.0), stream_unauthorized (stream 401 exhausted reconnects), and the voice connect-step codes (SDK 0.26.0) call_unsupported / call_connect_failed / call_dependency_missing on connectCall/startCall rejections. From SDK 0.28.1 the full list is drift-tested (SDK_SYNTHESIZED_ERROR_CODES).

9. End-to-end example — a game that plays along

1. app → hello { modalities:{text,call},
                 tools:[{ name:"game.highlight", description:"highlight a UI element",
                          parameters:{ type:"object", properties:{ target:{ type:"string" } } } }],
                 contextKinds:["game.player","game.event.*"] }
   (`tools[]` is what makes companion.tool_call possible — `handles[]` alone
    does not register a tool server-side)
   ←  hello.ack { session, grantedScopes, modalities, resumeCursor }

2. app → control.start_call
   ←  control.call_ready { provider:"elevenlabs-convai", agentId:"…" }  (secret-free)
   app opens WebRTC to ElevenLabs.

3. during play:
   app → context.event { type:"game.player.hp", retained:true, ttl:5000,
                         data:{ hp:80 } }                 // coalesced, ambient
   app → context.event { type:"game.event.boss_spawned", salience:0.9,
                         voiceRelevant:true, data:{ boss:"Dragon" } }

4. server pushes a contextual update into the live voice session →
   companion SAYS: "Boss incoming — heal up first!"

5. companion → companion.tool_call { id:"tc_1", name:"game.highlight",
                                     args:"{\"target\":\"healPotion\"}" }
   (`args` is the model's JSON **string** — parse it, or use the SDK's
   pre-parsed `argsJson`)
   app highlights the potion, then:
   app → tool.result { callId:"tc_1", ok:true }

The tighter steps 3-4 are wired (salience, coalescing, voiceRelevant → injection), the better the companionship feels — which is exactly why this plane gets the most care.