Pouchy Companion — Native (Android / iOS) + Matrix bridge integration

For a native app (Kotlin / Swift). The @pouchy_ai/companion-sdk npm package is JavaScript (browser + Node) and cannot be imported into native code — but it's only a thin convenience wrapper over a plain HTTPS REST + Server-Sent Events contract, which a Kotlin or Swift client implements directly. This guide is that contract, with reference clients you can drop in, plus a Matrix 代聊 bridge for letting a user's companion field their Matrix DMs on their behalf.

The wrapper SDK's behaviour is the spec: see src/lib/companion-sdk/client.ts. The full HTTP reference is companion-api-reference.md.

0. The shape of it

Two integration points, deliberately separate:

The bridge is best server-side (always-on, listens to Matrix). It can reuse the JS SDK directly (Node), so §4 is JS; §2–§3 are the native clients for the app itself.

1. Contract basics

• Base URL: https://pouchy.ai
• Auth: every call carries Authorization: Bearer pchy_… (a Personal Access Token the user mints via POST /api/companion/keys while signed in — no in-app Wallet panel for this yet — or an OAuth token). The SSE stream also accepts the token as a ?access_token= query param (for SSE clients that can't set headers).
• Scopes the token needs for this integration: chat, events.subscribe (receive replies), voice/call (optional, voice), and — for 代聊 — represent (+ optional expose:knowledge, expose:facts, represent:remember, represent:pair). Scopes are chosen at token creation; everything is gated server-side.
• Transport: REST for everything you send; one SSE stream for everything the companion sends back (its reply lands as a companion.message event, not in the POST response).

The core loop

POST /api/companion/session                         → { session, resumeCursor, … }
GET  /api/companion/session/{session}/stream?cursor=0   (open SSE, keep it open)
POST /api/companion/session/{session}/input  {text}     (the reply arrives on the stream)
   ← event: companion  data: {"type":"companion.message","payload":{"text":"…"}}

Endpoints used here

SSE wire format (exact)

The stream sends standard SSE frames. Each frame has an event: name and a JSON data: line. It runs ~45s then closes cleanly with a reconnect frame carrying the next cursor — reconnect with ?cursor=<that> (this is normal, not an error).

event: open
data: {"type":"hello.ack","payload":{"resumeCursor":12},"session":"sess_…"}

event: companion
data: {"v":1,"id":"evt_…","ts":1718900000,"type":"companion.message","payload":{"text":"Hi!"}}

: ping            ← heartbeat comment, ignore

event: reconnect
data: {"cursor":13}     ← reopen the stream with ?cursor=13

• event: open — first frame; payload.resumeCursor is where you are.
• event: companion — a companion event. Switch on data.type: companion.message (payload.text — the reply), companion.tool_call, companion.confirm_request, control.error (payload.code/message), …
• event: reconnect — bounded-lifetime close; immediately reopen with ?cursor=data.cursor. Track the highest cursor you've seen and pass it on reconnect so you never miss or duplicate an event.

2. Android — Kotlin reference client

Uses OkHttp for REST and okhttp-sse for the stream.

// build.gradle: implementation("com.squareup.okhttp3:okhttp:4.12.0")
//               implementation("com.squareup.okhttp3:okhttp-sse:4.12.0")
import okhttp3.*
import okhttp3.sse.*
import org.json.JSONObject

class PouchyCompanion(
    private val baseUrl: String,
    private val token: String,
    private val surface: String = "android",
    /** Pass for 代聊 / representative mode; null for the user's own companion. */
    private val visitor: Pair<String, String?>? = null   // (id, displayName?)
) {
    private val http = OkHttpClient.Builder()
        .readTimeout(0, java.util.concurrent.TimeUnit.MILLISECONDS) // SSE: no read timeout
        .build()
    private val json = "application/json".toMediaType()
    private var session: String? = null
    private var cursor: Int = 0
    private var es: EventSource? = null

    private fun req(path: String) = Request.Builder()
        .url(baseUrl.trimEnd('/') + path)
        .header("Authorization", "Bearer $token")

    /** Handshake. Returns the session id; populates representative/visitorPaired. */
    fun connect(): JSONObject {
        val body = JSONObject().put("surface", surface)
        visitor?.let { (id, name) ->
            body.put("visitor", JSONObject().put("id", id).apply { name?.let { put("displayName", it) } })
        }
        http.newCall(req("/api/companion/session").post(body.toString().toRequestBody(json)).build())
            .execute().use { res ->
                val ack = JSONObject(res.body!!.string())
                check(res.isSuccessful && ack.optBoolean("ok")) { "connect failed: ${ack.optString("error")}" }
                session = ack.getString("session")
                cursor = ack.optInt("resumeCursor", 0)
                return ack   // { session, representative, visitorPaired, grantedScopes, … }
            }
    }

    /** Open the auto-reconnecting event stream. onMessage receives reply text. */
    fun start(onMessage: (String) -> Unit, onError: (String) -> Unit = {}) {
        val s = session ?: error("call connect() first")
        // Header-based auth works for OkHttp SSE (unlike browser EventSource).
        val request = req("/api/companion/session/$s/stream?cursor=$cursor").build()
        es = EventSources.createFactory(http).newEventSource(request, object : EventSourceListener() {
            override fun onEvent(es: EventSource, id: String?, type: String?, data: String) {
                when (type) {
                    "reconnect" -> {                       // bounded close → reopen
                        cursor = JSONObject(data).optInt("cursor", cursor)
                        start(onMessage, onError)
                    }
                    "open" -> cursor = JSONObject(data).optJSONObject("payload")?.optInt("resumeCursor", cursor) ?: cursor
                    "companion" -> {
                        val env = JSONObject(data)
                        if (env.optString("type") == "companion.message")
                            onMessage(env.getJSONObject("payload").optString("text"))
                        if (env.optString("type") == "control.error")
                            onError(env.getJSONObject("payload").optString("message"))
                    }
                }
            }
            override fun onFailure(es: EventSource, t: Throwable?, res: Response?) {
                // 401/403 is permanent (missing events.subscribe / revoked) — surface it.
                if (res?.code == 401 || res?.code == 403) onError("stream unauthorized (${res.code})")
                else { Thread.sleep(1500); start(onMessage, onError) }   // transient → backoff + retry
            }
        })
    }

    fun stop() { es?.cancel(); es = null }

    /** Send a user turn. The reply arrives via the onMessage you passed to start(). */
    fun sendText(text: String) {
        val s = session ?: error("call connect() first")
        val body = JSONObject().put("text", text).toString().toRequestBody(json)
        http.newCall(req("/api/companion/session/$s/input").post(body).build()).execute().close()
    }
}

Usage (the user's own companion):

val c = PouchyCompanion(baseUrl = "https://pouchy.ai", token = userPat)
c.connect()
c.start(onMessage = { reply -> runOnUiThread { showBubble(reply) } })
c.sendText("How's my day looking?")

3. iOS — Swift reference client

URLSession for REST + an URLSession.bytes line reader for SSE (no third-party dependency). Swift Concurrency (async/await).

import Foundation

struct CompanionAck: Decodable { let session: String; let resumeCursor: Int?
    let representative: Bool?; let visitorPaired: Bool? }

final class PouchyCompanion {
    let baseURL: URL; let token: String; let surface: String
    let visitor: (id: String, displayName: String?)?     // pass for 代聊 mode
    private var session: String?; private var cursor = 0
    private var streamTask: Task<Void, Never>?

    init(baseURL: String, token: String, surface: String = "ios",
         visitor: (id: String, displayName: String?)? = nil) {
        self.baseURL = URL(string: baseURL)!; self.token = token
        self.surface = surface; self.visitor = visitor
    }

    private func request(_ path: String, method: String = "GET", body: [String: Any]? = nil) -> URLRequest {
        var r = URLRequest(url: baseURL.appendingPathComponent(path))
        r.httpMethod = method
        r.setValue("Bearer \(token)", forHTTPHeaderField: "Authorization")
        if let body { r.setValue("application/json", forHTTPHeaderField: "Content-Type")
            r.httpBody = try? JSONSerialization.data(withJSONObject: body) }
        return r
    }

    @discardableResult
    func connect() async throws -> CompanionAck {
        var body: [String: Any] = ["surface": surface]
        if let v = visitor {
            var vo: [String: Any] = ["id": v.id]; if let n = v.displayName { vo["displayName"] = n }
            body["visitor"] = vo
        }
        let (data, _) = try await URLSession.shared.data(for: request("api/companion/session", method: "POST", body: body))
        let ack = try JSONDecoder().decode(CompanionAck.self, from: data)
        session = ack.session; cursor = ack.resumeCursor ?? 0
        return ack
    }

    /// Open the auto-reconnecting SSE stream. `onMessage` fires per reply.
    func start(onMessage: @escaping (String) -> Void, onError: @escaping (String) -> Void = { _ in }) {
        guard let s = session else { return }
        streamTask = Task {
            while !Task.isCancelled {
                do {
                    let req = request("api/companion/session/\(s)/stream?cursor=\(cursor)")
                    let (bytes, resp) = try await URLSession.shared.bytes(for: req)
                    if let code = (resp as? HTTPURLResponse)?.statusCode, code == 401 || code == 403 {
                        onError("stream unauthorized (\(code))"); return        // permanent
                    }
                    var event = "message"
                    for try await line in bytes.lines {
                        if line.hasPrefix("event:") { event = line.dropFirst(6).trimmingCharacters(in: .whitespaces) }
                        else if line.hasPrefix("data:") {
                            let json = String(line.dropFirst(5)).trimmingCharacters(in: .whitespaces)
                            handle(event: event, data: json, onMessage: onMessage, onError: onError)
                        }
                        // blank line = frame boundary; ": ping" comments are ignored
                    }
                } catch { try? await Task.sleep(nanoseconds: 1_500_000_000) }  // transient → backoff
            }
        }
    }

    private func handle(event: String, data: String, onMessage: (String) -> Void, onError: (String) -> Void) {
        guard let obj = try? JSONSerialization.jsonObject(with: Data(data.utf8)) as? [String: Any] else { return }
        switch event {
        case "reconnect": cursor = obj["cursor"] as? Int ?? cursor          // loop reopens with new cursor
        case "open": cursor = (obj["payload"] as? [String: Any])?["resumeCursor"] as? Int ?? cursor
        case "companion":
            let type = obj["type"] as? String
            let payload = obj["payload"] as? [String: Any]
            if type == "companion.message", let text = payload?["text"] as? String { onMessage(text) }
            if type == "control.error", let msg = payload?["message"] as? String { onError(msg) }
        default: break
        }
    }

    func stop() { streamTask?.cancel(); streamTask = nil }

    func sendText(_ text: String) async throws {
        guard let s = session else { return }
        _ = try await URLSession.shared.data(for: request("api/companion/session/\(s)/input", method: "POST", body: ["text": text]))
    }
}

Note on the reconnect cursor: URLSession.bytes keeps one connection open; when the server closes at ~45s the for try await … lines loop ends, the while re-enters, and it reopens with the cursor the reconnect frame set. Track the highest cursor seen so a reconnect never replays or skips an event.

4. Matrix 代聊 bridge (the companion answers the user's DMs)

Goal: when someone DMs user A in your Matrix-based IM and A wants their companion to field it, the companion replies on A's behalf — screened (never A's private memory), with per-visitor continuity. This is exactly representative mode; the bridge maps Matrix ⇄ the representative API.

Architecture

Run a small server-side bridge (Node) — always-on, holds each opted-in user's PAT (with represent + chat + events.subscribe), and a Matrix bot/appservice identity to read the room and post replies.

Matrix DM room ──(m.room.message from visitor B)──▶ bridge
   bridge: representative session  (token = A's PAT, visitor = B)
        POST /session {visitor:{id:enc(B), displayName:B}}  → session
        POST /input {text: B's message}
        ◀ SSE companion.message {text}                       (A's companion, screened)
   bridge ──(post reply as A / as a puppet)──▶ Matrix DM room

Mapping rules

• visitor.id must match ^[A-Za-z0-9_-]{8,64}$. A Matrix MXID (@bob:server.org) isn't url-safe, so encode it stably — e.g. a hex SHA-256, truncated. The same B always maps to the same id → continuous per-visitor thread
  • memory.
• surface = matrix:<roomId> — one resumable session per (A, room).
• displayName = B's Matrix display name (shown to A's companion).
• One bridge process can multiplex many users; key sessions by (A's token, surface).

Reference (Node — the JS SDK works server-side)

// npm i @pouchy_ai/companion-sdk matrix-bot-sdk
import { createCompanion } from '@pouchy_ai/companion-sdk';
import { MatrixClient, SimpleFsStorageProvider, AutojoinRoomsMixin } from 'matrix-bot-sdk';
import { createHash } from 'node:crypto';

const BASE = 'https://pouchy.ai';
const encVisitor = (mxid: string) => createHash('sha256').update(mxid).digest('hex').slice(0, 48);

// Your app maps a Matrix user → their Pouchy PAT (and whether 代聊 is enabled).
declare function ownerForRoom(roomId: string): Promise<{ ownerMxid: string; pouchyPat: string } | null>;

const matrix = new MatrixClient(HOMESERVER_URL, BOT_ACCESS_TOKEN, new SimpleFsStorageProvider('bridge.json'));
AutojoinRoomsMixin.setupOnClient(matrix);

matrix.on('room.message', async (roomId, ev) => {
  const sender = ev.sender as string;
  const body = ev.content?.body as string | undefined;
  if (!body || ev.content?.msgtype !== 'm.text') return;

  const owner = await ownerForRoom(roomId);
  if (!owner || sender === owner.ownerMxid) return;        // only field messages TO the owner

  // Representative session: companion speaks for the owner, to this visitor.
  const c = createCompanion({
    baseUrl: BASE,
    token: owner.pouchyPat,                                 // owner's PAT (holds `represent`)
    surface: `matrix:${roomId}`,
    appContext: { name: 'YourApp DMs', description: 'Matrix direct message' },
    visitor: { id: encVisitor(sender), displayName: (await displayName(matrix, sender)) }
  });
  await c.connect();                                        // → { representative: true, … }

  // Get the one reply for this turn, post it back, then close.
  const reply = await new Promise<string>((resolve) => {
    const off = c.onMessage((t) => { off(); resolve(t); });
    c.start();
    c.sendText(body).catch(() => resolve(''));
  });
  if (reply) await matrix.sendText(roomId, reply);          // post as the bot/owner puppet
  c.stop();
});

await matrix.start();

Optional: pairing two Pouchy users

If both A and B use Pouchy, the bridge can pair them so their companions become friends (unlocking the A2A plane in their own apps). When B hands the bridge their own PAT (holding social.message):

await c.pairVisitor(visitorPat);     // owner PAT holds `represent:pair`

Privacy invariants the bridge inherits (free)

• A's companion only ever reveals screened context to B (persona + safe facts + — if A granted expose:knowledge — shared materials). A's system prompt, private memory, intimate facts, and PII are never exposed.
• B's messages are quarantined: they don't enter A's personal memory. With represent:remember, the bridge gets durable per-visitor notes (in an isolated store) so a returning B is remembered.
• A token without represent is rejected (403) — the bridge can't accidentally open an owner-facing session against a stranger.

5. Voice (native, WebRTC direct-to-provider)

Pouchy only mints a short-lived credential; the audio is a WebRTC session directly between the device and the voice provider (ElevenLabs or OpenAI) — the Pouchy server is out of the audio path, which is what makes it cheap + low-latency. So the native job is: (1) call the mint endpoint (REST, trivial), then (2) hand the returned creds to the provider's own native path.

5.1 Mint

POST /api/companion/session/{session}/call
  body: { voice?, locale? }            (locale drives voice + language pick)
  → ElevenLabs:  { ok, provider:"elevenlabs-convai", token, agentId, instructions, voice, language, firstMessage }
  → OpenAI:      { ok, provider:"openai-realtime",  clientSecret, model, voice, expiresAt }

Needs the call scope. The server picks the provider (Convai primary → OpenAI fallback). In a representative session the instructions are already screened + visitor-facing — nothing extra to do; you just open the call.

5.2 ElevenLabs Convai (primary — the user's cloned voice)

Use ElevenLabs' native Conversational-AI SDK (Android + iOS, see the ElevenLabs docs) — it owns the mic/WebRTC. Start a session with our minted values mapped to its overrides:

Kotlin sketch (mint with the §2 client, then the EL SDK does the audio):

val creds = JSONObject(/* POST .../call response */)
if (creds.getString("provider") == "elevenlabs-convai") {
    // ElevenLabs Android Conversational-AI SDK owns mic + WebRTC. Pass:
    //   conversationToken = creds.getString("token")
    //   overrides.agent.prompt.prompt = creds.getString("instructions")
    //   overrides.tts.voiceId        = creds.getString("voice")
    //   overrides.agent.language     = creds.optString("language")
    //   overrides.agent.firstMessage = creds.optString("firstMessage")
    // (exact API per the ElevenLabs SDK version you depend on)
}

5.3 OpenAI Realtime (fallback — zero extra SDK, standard WebRTC)

Open a WebRTC peer connection straight to OpenAI with the ephemeral clientSecret. This is the standard OpenAI Realtime WebRTC handshake — works with any platform WebRTC lib (Android: org.webrtc / io.getstream:stream-webrtc-android; iOS: the WebRTC / GoogleWebRTC pod).

1. Create RTCPeerConnection; addTrack(local mic audio).
2. offer = pc.createOffer(); pc.setLocalDescription(offer)
3. POST https://api.openai.com/v1/realtime?model={creds.model}
     Authorization: Bearer {creds.clientSecret}
     Content-Type: application/sdp
     body: offer.sdp
   → 200, body = answer SDP
4. pc.setRemoteDescription({ type:"answer", sdp: answerBody })
5. Audio flows. Use a data channel for events (transcripts, tool calls) if needed.

expiresAt bounds the credential — mint per call. Instructions (screened, in representative mode) are baked into the session server-side, so nothing to inject.

5.4 Memory after a voice call

The audio never touches Pouchy, so to let the companion remember a call, POST the transcript on close (the device has it from the provider):

POST /api/companion/session/{session}/end   body: { transcript: [{role,text}, …] }

For a representative voice call this is quarantined automatically — the transcript never enters the owner's memory; it distils into per-visitor notes only under represent:remember.

Deeper background (provider precedence, world-state injection mid-call, the first-party call): companion-voice-integration.md.

6. Avatar (native, VRM)

GET /api/companion/avatar → the user's current companion look, so you render the same virtual human Pouchy shows:

GET /api/companion/avatar
  → { ok, name, archetype:"girlfriend"|"boyfriend"|"pet", modelId, vrmUrl, imageUrl }

Any valid token (no special scope). vrmUrl is a VRM model (a glTF 2.0 extension) on a CORS-enabled host; imageUrl is a flat 2D portrait when one exists (often null for built-ins), useful as a lightweight avatar image / fallback.

Native rendering options, simplest first:

• 2D portrait — if imageUrl != null, just load it into an ImageView / UIImageView. Zero 3D work; good for chat bubbles, lists, call screens.
• 3D VRM in a WebView (pragmatic full 3D) — host a tiny page using @pixiv/three-vrm that takes ?vrm=<vrmUrl>, and drop it in an Android WebView / iOS WKWebView. The /models asset is CORS-enabled so the in-WebView fetch works. This reuses the mature web VRM stack and gets you expressions/lip-sync hooks for free.
• Native 3D engine — load the glTF/VRM directly: Android with Filament or SceneView; iOS with GLTFKit/SceneKit or RealityKit. VRM-specific features (spring-bone, blendshape presets) need a VRM-aware loader — budget for it, or start with the WebView path.

// REST part is trivial with the §2 client's req():
http.newCall(req("/api/companion/avatar").build()).execute().use { res ->
    val a = JSONObject(res.body!!.string())
    val vrmUrl = a.optString("vrmUrl")     // feed your renderer / WebView
    val imageUrl = a.optString("imageUrl", "")  // 2D fallback
}

getAvatar reflects the user's live choice (built-in or a custom upload) and falls back to the archetype default if unset.

7. Notes

• Bluesky / AT Protocol (your social layer) is orthogonal to the companion and has its own spec — DMs (代聊), 代读/代发, and social-graph→pairing — in companion-bluesky-bridge.md. Same companion API, a different protocol adapter.
• The JS wrapper (@pouchy_ai/companion-sdk) is the canonical behaviour reference for every REST/SSE flow above — read src/lib/companion-sdk/client.ts when a detail is ambiguous, and the full HTTP contract is in companion-api-reference.md.
• All /api/companion/* responses are CORS-enabled; the SSE stream is bounded (~45s) by design — reconnect with ?cursor= is the normal path, not an error.