Companion host-control verbs (universal app actions)

A small set of generic verbs that let the companion act in ANY surface — web, game, app, hardware — through one onToolCall handler. They're intentionally generic: the specific intent rides in the string params, which your surface interprets. That keeps the schema fixed (so it can be pre-registered on the otherwise-static ElevenLabs Convai agent) while still covering arbitrary actions.

Exported from the SDK as HOST_CONTROL_TOOLS (the JSON-Schema definitions) and HOST_CONTROL_TOOL_NAMES.

Where they're offered — voice calls only, unless you declare them. The SDK adds these verbs automatically to connectCall() voice sessions when your embed declares tools or handles. Text turns are different: the server offers the model exactly the tools[] your session declared — nothing is auto-added — so a text embed that wants the verbs on text turns must declare them itself (createCompanion({ tools: [...HOST_CONTROL_TOOLS] })). A declared verb then flows like any other tool: the model calls it, companion.tool_call is emitted, your onToolCall fires. (handles[] is stored but never consumed server-side today — treat it as a client-side capability hint; its one SDK effect is opting a tool-less embed into the auto-offered verbs on voice calls. A tool the app wants called on text turns MUST be declared in tools[].)

The verbs

Handle them (one onToolCall handler)

companion.onToolCall(async ({ id, name, args }) => {
  const a = JSON.parse(args);
  switch (name) {
    case 'pause':            // invoke_action({action:'pause'}) → name:'pause'
    case 'pause_game': game.pause(); break;
    case 'resume': game.resume(); break;
    case 'set_feature': ui.toggle(a.feature, a.on); break;
    case 'set_value':  ui.set(a.name, a.value); break;
    case 'navigate':   router.go(a.to, a.params); break;
    case 'highlight':  ui.highlight(a.target); break;
  }
  await companion.sendToolResult(id, { ok: true });
});

invoke_action arrives unwrapped: invoke_action({action:'pause_game', params}) fires onToolCall with name:'pause_game', args: params — so it hits the same handler a declared pause_game tool would. The other verbs pass through by name.

The same handler serves both planes: voice-call verbs arrive here via the SDK's call bridge, and text-turn tool calls arrive here via companion.tool_call — but remember the verbs only reach a text turn if you declared them in tools[] (see the offering rules above).

Benign actions (above) run immediately. Only sensitive ops (wallet.spend / social.message / skills.execute) go through the Pouchy confirm boundary.

Provider note

• OpenAI Realtime: the SDK registers these (and any declared tools) dynamically via session.update — works out of the box.
• ElevenLabs Convai: a shared Convai agent only calls tools it's been configured with, so these verbs must be registered once on the agent (below). Until then, voice tool-calling on the EL path won't fire (the OpenAI voice path is unaffected; text turns only ever see the tools your session declared, per the offering rules above).

One-time ElevenLabs Convai agent registration

Add these as client tools on the companion Convai agent (Dashboard → the agent → Tools → Add client tool, or via the EL API). Name + description + parameters must match HOST_CONTROL_TOOLS:

[
  { "name": "invoke_action",
    "description": "Trigger a named action the current surface supports — e.g. pause, resume, start, stop, restart, next, previous, submit, confirm, cancel, refresh. Put the action name in \"action\" and any extra args in \"params\".",
    "parameters": { "type": "object", "properties": {
      "action": { "type": "string" }, "params": { "type": "object" } }, "required": ["action"] } },
  { "name": "set_feature",
    "description": "Turn a named feature/capability on or off (hints, captions, dark mode, a light, notifications, a mode).",
    "parameters": { "type": "object", "properties": {
      "feature": { "type": "string" }, "on": { "type": "string" } }, "required": ["feature", "on"] } },
  { "name": "set_value",
    "description": "Set a named control to a value — volume, brightness, difficulty, speed, zoom, temperature, etc.",
    "parameters": { "type": "object", "properties": {
      "name": { "type": "string" }, "value": { "type": "string" } }, "required": ["name", "value"] } },
  { "name": "navigate",
    "description": "Go to a named view / screen / page / section / mode / level in the surface.",
    "parameters": { "type": "object", "properties": {
      "to": { "type": "string" }, "params": { "type": "object" } }, "required": ["to"] } },
  { "name": "highlight",
    "description": "Draw the user's attention to a named element (guidance / onboarding / a tutorial step).",
    "parameters": { "type": "object", "properties": {
      "target": { "type": "string" }, "note": { "type": "string" } }, "required": ["target"] } }
]

The SDK provides the handlers for these at call time (clientTools); the agent config above provides the schemas so the model knows it may call them. The result the app reports via sendToolResult is fed back to the voice model.

ElevenLabs client-tool params are always strings, so set_feature.on is declared as a string ("true"/"false"). The SDK normalizes it to a real boolean before calling your onToolCall, so your handler reads a.on as a boolean on every provider — no per-provider branching.

Avatar emote tools (play_gesture / play_expression)

The embodied ("形象") companion can also call two avatar visual tools to emote — play_gesture (full-body animation) and play_expression (facial expression). Exported from the SDK as AVATAR_VISUAL_TOOLS / AVATAR_VISUAL_TOOL_NAMES. Like the host verbs their schemas are registered on the shared Convai agent (a manual EL-console step — see the registration section below), so the companion can call them on every voice call.

On voice calls you do not need to declare or handle them. The SDK offers them on every connectCall() session — even a pure-chat embed that declared no tools — and answers with a silent no-op success ({ ok: true }) for any embed that didn't opt in, so a pure audio or 2D integrator can ignore avatars entirely and the companion's gesture/expression calls simply do nothing instead of erroring. (On the text plane the question doesn't arise: the server offers only the session's declared tools[], so an undeclared avatar tool is never called on a text turn.)

To actually render the avatar, opt in by declaring the same tool name in createCompanion({ tools }) (your own schema wins) and handling it in onToolCall just like any other tool — the SDK then routes the call to you instead of no-oping, and the declaration also puts the tool on your text turns:

createCompanion({
  // ...
  tools: [
    { name: 'play_gesture',    description: 'play an avatar gesture',
      parameters: { type: 'object', properties: { category: { type: 'string' } }, required: ['category'] } },
    { name: 'play_expression', description: 'set the avatar expression',
      parameters: { type: 'object', properties: { intent:   { type: 'string' } }, required: ['intent'] } }
  ]
});
companion.onToolCall(async ({ id, name, args }) => {
  const a = JSON.parse(args);
  if (name === 'play_gesture') avatar.playGesture(a.category);
  if (name === 'play_expression') avatar.setExpression(a.intent);
  await companion.sendToolResult(id, { ok: true });
});

How registration actually works today

Two halves, per call vs. per agent:

• Per call (automatic): at connectCall() the SDK passes a handler function for every offered tool into the EL session's clientTools map (its VoiceToolBridge, see src/lib/companion-sdk/call.ts). The first-party app does the same in src/lib/services/convai/connection.ts, whose runners come from src/lib/engine/call-tools.ts. On OpenAI Realtime the schemas ALSO ride per call (session.update), so that path needs nothing else.
• Per agent (manual, EL only): ElevenLabs only lets the model call a client tool the agent's own config already declares, and the shared Convai agent's config is static — so the schema pre-registration in the section above is a manual, one-time ElevenLabs-console step (Dashboard → the agent → Tools → Add client tool, or the EL API). The old Ops-dashboard "ElevenLabs Convai tool sync" panel and its DESIRED_CONVAI_TOOLS constant have been removed — there is no in-app sync tool anymore. After adding a NEW verb to HOST_CONTROL_TOOLS / AVATAR_VISUAL_TOOLS, register its schema on the agent by hand.