Made in 10Docs
GuidesAPI
Real time

The sound socket

The WebSocket a person's browser or app joins for a call's sound: connecting, the messages both ways, and what to get right.

A call's sound travels over one WebSocket between the person's browser (or app) and us: their microphone up, the AI's voice down. Its address is the ws_url that starting a call answered.

Use the SDK

Our browser SDK speaks this protocol for you: the microphone, echo cancellation, interruptions and every kind of face. Read on if you are building your own client, for example a native app.

Connecting

ws_url
wss://api.daxov.com/live/<session>?token=<token>
  • Connect within 120 seconds of starting the call.
  • The token is the login. It names one call, works once, and expires 5 minutes after it was made.
  • A refused connection is accepted and then closed at once, because a browser can't read why an upgrade was refused. The close code says why:
Close codeWhy
4401The token is wrong or has expired
4409Someone already connected with it, or no such call is waiting

The call keeps waiting for a good connection until its 120 seconds are up.

Frames

Binary frames carry sound; text frames carry JSON. Sound is 16-bit little-endian PCM, mono, at the rate agreed in the first two messages. A frame may be any length; 20 ms is ideal (640 bytes at 16 kHz).

From the browser

MessageWhen
{"type":"hello","v":1,"input":{"format":"pcm16","rate":16000}}First, once. Sound sent before it is ignored
binaryThe person's microphone, at input.rate (the CallStart's ws.input_rate)
{"type":"played","mark":7}All the AI's sound sent before mark 7 has finished playing
{"type":"bye"}The person ended the call

To the browser

MessageWhen
{"type":"ready","output":{"format":"pcm16","rate":24000}}After the hello: the call is starting
binaryThe AI's voice, at output.rate. Play it in order, as it comes
{"type":"mark","mark":7}The end of one AI reply. Answer with played once it has finished playing
{"type":"clear"}The person interrupted: drop all AI sound not yet played, at once
{"type":"status","status":"running"}The call's status changed
{"type":"event","text":"…"}A line of the call's log, for your own logs
{"type":"ended","reason":"…","code":"time_limit"}The call is over; then the socket closes. code is short and stable: time_limit, person_left, ended, silence, never_connected, unavailable, error

We send a ping every 15 seconds, so proxies keep the connection open. Ignore message types you don't know: more may come.

The website widget's own call page also sends "page" in its hello and answers tool_call with tool_result, for tools on the visitor's page. That is only for a call in the bubble; your own client never needs it.

The shape of a client
const ws = new WebSocket(start.ws_url);
ws.binaryType = "arraybuffer";

ws.onopen = () => {
  ws.send(JSON.stringify({ type: "hello", v: 1, input: { format: "pcm16", rate: start.ws.input_rate } }));
  // then send the microphone as binary 16-bit PCM frames at that rate
};

ws.onmessage = ({ data }) => {
  if (data instanceof ArrayBuffer) return queueForPlayback(data); // the AI's voice
  const msg = JSON.parse(data);
  if (msg.type === "mark") whenPlayedUpTo(msg.mark, () => ws.send(JSON.stringify({ type: "played", mark: msg.mark })));
  if (msg.type === "clear") dropEverythingNotYetPlayed();
  if (msg.type === "ended") showEnded(msg.reason);
};

function hangUp() {
  ws.send(JSON.stringify({ type: "bye" }));
  ws.close();
}

What the browser must get right

  • The microphone: ask for it with echo cancellation, noise suppression and automatic gain on (getUserMedia({ audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true } })), and send it at the rate you declared in the hello, 16 kHz.
  • Echo: the AI's voice must not reach the microphone, or the AI hears itself and stops talking. Play it so that the browser's echo cancellation sees it. Headphones avoid echo whichever way.
  • Interruptions: clear drops everything queued, at once. A reply cut off this way is never answered with played.
  • Marks: answer every mark that played to the end, in order. They time each turn.
  • Ending: send bye, then close. Closing without it (a tab closed, the network gone) ends the call too.

With a face

For a character with a face, the CallStart's ws.face says where the face plays and how to feed it, and output_rate is the face's own rate. The browser then:

  1. joins the face's room and shows its video. The face's sound is the AI's voice, in sync with its lips: play that;
  2. still sends the microphone up this socket, as above;
  3. forwards our binary frames to the face instead of playing them, and tells us when the face speaks: {"type":"face","speaking":true}, then false once it's quiet.

ws.face.vendor is always livekit: every face is one of ours and plays in a room on our own media server, which speaks the open-source LiveKit protocol. The SDK does all of this. Without it, join the room with any LiveKit client library and feed the face as the CallStart's FaceTicket describes: livekit_url and token to join, avatar_identity to watch and feed, and our voice as byte streams on the topic lk.audio_stream.

When things go wrong

WhatWhat happens
The browser never connectsAfter 120 seconds the call ends, never_connected. Nothing is charged
The token is wrong, expired or usedClosed at once with 4401 or 4409; the call keeps waiting for a good one
The connection drops mid-callThe call ends: there is no reconnecting yet
Something fails on our sideended with a reason, then the socket closes
The browser sends sound at another rate than its hello saidThe AI hears it wrong. Always send the rate you declared

On this page