Companion SDK — Integrator Quickstart

Embed the user's Pouchy companion in your app/game/site — it arrives knowing them (memory, name, preferences), can chat by text or voice, see images, act in your app, and react in real time to what's happening. This is the hands-on guide; the wire spec is companion-sdk-protocol.md.

Want to see every call working against the live backend first? The in-app playground (/app/admin/timing#playground) drives every plane end-to-end and shows the raw wire — it's the reference implementation for everything below.

1. Get a token (30 seconds)

Standard path (developers / apps with their own users): create a project in the dashboard, mint a Secret Key (dashboard → Keys), and have YOUR BACKEND exchange it for a per-user session token — first-seen external_user_ids are provisioned automatically, so your users never configure anything:

POST https://pouchy.ai/v1/sessions
Authorization: Bearer pchy_sk_…
{ "agent": "<agentId>", "external_user_id": "user_4211" }

→ { "session_token": "pchy_…", "expires_in": 3600, "instance": { … } }

The session_token is the token every snippet below uses. Two end-to-end walkthroughs of this flow: quickstart-romance-companion.md and quickstart-commerce-guide.md.

Advanced path (personal / single-user integrations): the user generates a Personal Access Token via POST /api/companion/keys while signed in to pouchy.ai (no in-app Wallet panel for this yet), picking the scopes + modalities they're comfortable granting (omit scopes for the safe default pack), and pastes it into your app. One token = one user = the same companion + memory everywhere. The rest of this guide works identically with either token.

Authorization: Bearer pchy_<secret>

Scopes you'll commonly want (off unless the user grants them):

A token without a scope simply can't reach that capability — start minimal.

2. Install & connect

import { createCompanion } from '@pouchy_ai/companion-sdk'; // or src/lib/companion-sdk

const companion = createCompanion({
  baseUrl: 'https://pouchy.ai',
  token: PCHY_TOKEN,
  surface: 'my-rpg',          // one resumable session per surface
  modalities: ['text', 'call', 'files'],
  contextKinds: ['game.player.*', 'game.event.*'], // what you'll stream (§4)
});

const ack = await companion.connect();   // { session, grantedScopes, modalities, resumeCursor }
companion.start();                        // open the auto-reconnecting event stream

companion.onMessage((text) => render(text));          // assistant replies
await companion.sendText('hey, I just logged in');     // → reply via onMessage

start() needs events.subscribe; without it replies never arrive (the SDK emits a control.error { code: 'stream_unauthorized' } so you notice). Call stop() to close the stream.

3. The five planes

Text + images

await companion.sendText('what do you think of this?', { images: [dataUrl] }); // needs `files`
companion.onMessage((text) => …);

Downscale photos client-side (≈1024px JPEG) before sending — raw camera images blow past the request-body limit.

Voice call

const creds = await companion.startCall({ locale: navigator.language });
// creds is discriminated by provider — open WebRTC directly to it:
//  'elevenlabs-convai' → { token, agentId, instructions, voice?, language? }
//  'openai-realtime'   → { clientSecret, model, voice, expiresAt }

Pouchy stays out of the audio path; you connect peer-to-peer to the provider. Pass locale so the call opens in the user's language with their cloned voice. See the playground's startVoiceCall for a working connect using the bundled wrappers.

Tools — let the companion act in your app

createCompanion({ …, tools: [{
  name: 'give_item',
  description: 'Give the player an item.',
  parameters: { type: 'object', properties: { item: { type: 'string' } }, required: ['item'] },
}]});

companion.onToolCall(async ({ id, name, argsJson }) => {
  if (name === 'give_item') {
    grantItem((argsJson as { item: string }).item);   // argsJson = pre-parsed args
    await companion.sendToolResult(id, { ok: true, result: 'done' });
  }
});

The companion only ever calls a tool you declared and the token scoped. When all of a turn's tool results are in, it continues and replies.

Memory

const facts = await companion.recall({ limit: 8 });          // ranked, scope-filtered
await companion.remember({ content: 'Plays as a healer main', importance: 60 });

World-state — the companionship channel (§4 below)

companion.sendWorldState({ type: 'game.scene', data: 'rainy tavern', retained: true });

4. What to stream in (and how) — making it accompany well

This is the question that decides whether the companion feels present or oblivious. In bidirectional mode your app streams a live picture of what's happening; the companion reacts to it — in chat at the next turn, and (with voiceRelevant) out loud mid-call. Only type is required; the server fills the rest.

Two flavors

Knobs on a moment: salience (0–1, how much it should grab attention), voiceRelevant: true (push into a live call so it speaks now), ttl (ms a retained value stays fresh).

Signal menu by app type

Do / don't

• Do name kinds with a stable dotted namespace (game.player.hp) and declare them in contextKinds — it lets the server route ambient vs. trigger.
• Do mark the few genuinely interrupt-worthy moments voiceRelevant — that's the magic in a call ("boss incoming — heal!"). Keep it rare; it speaks over them.
• Do lean on retained for anything that changes often (position, hp, scrub position). It's coalesced — cheap.
• Don't stream raw per-frame telemetry as moments — it's throttled and wastes the user's quota; the server only ever injects a compact rolling digest, never the firehose.
• Don't put secrets/PII in data — it can reach the model and memory.

Minimal "plays along" loop

// ambient state — coalesced, consulted when relevant
companion.sendWorldState({ type: 'game.player.hp', data: { hp: 80 }, retained: true, ttl: 5000 });

// a moment worth reacting to, out loud, mid-call
companion.sendWorldState({
  type: 'game.event.boss_spawned',
  data: { boss: 'Dragon' },
  salience: 0.9,
  voiceRelevant: true,
});
// → in a call, the companion says: "Boss incoming — heal up first!"

5. Resilience

• Resume: the stream auto-reconnects from resumeCursor; missed outbound events replay by cursor, deduped by id. You don't manage this.
• Errors: failures surface as control.error { code, message } on the stream (subscribe via companion.on('control.error', …)), and REST calls reject with the message. A permanent stream auth failure stops the retry loop and emits stream_unauthorized rather than spinning silently.
• Start minimal: ship chat + events.subscribe first, confirm replies, then add worldstate.write, call, files, and the sensitive scopes as you need them.

6. Next

• Spec / message types: companion-sdk-protocol.md
• Scope vocabulary: src/lib/types/companion-scopes.ts
• Live reference (every call, raw wire): the playground at /app/admin/timing#playground