Protocol

Voice sessions

Live speech-to-speech with OpenAI Realtime models, billed per reply from the user's balance.

A voice session is a live, spoken conversation between your user and an OpenAI Realtime model (gpt-realtime-2.1, gpt-realtime-mini). Billing works like chat: each model reply is charged to the user’s Wind balance at provider cost plus Wind’s fee and your margin.

The audio itself never passes through Wind. The user’s browser talks to OpenAI directly over WebRTC, so Wind adds no latency. Wind watches the call from the server side and bills each reply. Nothing secret reaches the browser. It gets a single-use ticket that’s good for one offer exchange and nothing else.

How it fits together

1. Your server creates the session with the connection’s access token and passes session.client to the browser.
2. The browser connects with connectVoice(client) from @usewind/browser. That is the browser’s only call to Wind. After it, audio flows straight to OpenAI.
3. Your server attaches to the control channel (optional). Use it to answer tool calls, read per-reply receipts, steer the conversation and end the call.

1 · Create the session (server)

import { wind } from '@usewind/node'

app.post('/voice/start', async (req, res) => {
  const session = await wind
    .connection(req.user.id)
    .model('gpt-realtime-2.1')
    .realtime.create({ instructions: 'You are a friendly cooking coach.', voice: 'marin', tools })

  res.json(session.client) // { offer_url, ticket, expires_at } — safe for the browser
  watch(session)           // keep the control channel open after responding (step 3)
})

As with chat, you can pass feature instead of model to use the feature’s default_model. The model has to be in the app’s allowed_models. tools take the chat shape ({ type, function: { name, description, parameters } }) or the Realtime flat shape, and Wind converts between them. Creating the session also places the hold for the first reply. If that hold doesn’t fit the connection’s allocation, the call fails with 402 the same way a chat call does.

The ticket expires after about 60 seconds, so create the session when the user presses the call button, not ahead of time.

2 · Connect the browser

import { connectVoice } from '@usewind/browser'

const client = await (await fetch('/voice/start', { method: 'POST' })).json()
const call = await connectVoice(client) // asks for the mic, plays the model's audio

hangUpButton.onclick = () => call.hangUp()

Without the SDK, create an RTCPeerConnection, add the microphone track (or a recvonly audio transceiver), and create a data channel named oai-events before the offer. Then POST the SDP offer:

Send Content-Type: application/sdp with the offer as the body. The 201 response body is the SDP answer, which you pass to setRemoteDescription({ type: 'answer', sdp }). CORS is open to any origin. A ticket that’s wrong, expired or already used gets 401 invalid_ticket.

3 · The control channel (server)

session.control() opens the session’s control_url WebSocket (wss://api.usewind.app/v1/realtime/sessions/{id}/control, authenticated with the same access token) and yields one event per message.

async function watch(session) {
  for await (const ev of session.control()) {
    if (ev.type === 'response.function_call_arguments.done') {
      await session.toolResult(ev.call_id, await runTool(ev))
    }
    if (ev.type === 'wind.response.settled') console.log(ev.wind.user_charge_mc)
  }
}

toolResult sends the function_call_output item followed by a response.create, which lets the model continue speaking. Without a control channel, Wind answers every tool call with an error result so the conversation can keep going. To use tools you need the control channel.

Wind → your server

Wind also relays these OpenAI events: response.function_call_arguments.done (call_id, name, arguments), response.output_audio_transcript.done, response.output_text.done, input_audio_buffer.speech_started / speech_stopped, response.created and error.

Your server → Wind

session.update, conversation.item.create, response.create, response.cancel, conversation.item.truncate, { type: 'wind.session.end', when? } and { type: 'wind.session.approve_block' }. Any other event type gets back an error with code invalid_request. Your server is trusted, so the config it sets with session.update is what Wind enforces. Input transcription isn’t available.

If the browser changes instructions, voice or other session config, Wind puts them back, unless you listed them in client_updatable when creating the session (voice and instructions only). Set on_control_lost: "end" to end the call 10 seconds after your control channel drops. The default is "continue".

Close codes: 4004 means the session wasn’t found or isn’t yours. 4009 means the session isn’t live.

Ending a session

await session.end({ when: 'immediate' })

• after_heard (default): waits until the user has heard the current reply, then hangs up.
• immediate: cancels the reply in progress, bills the part already generated, then hangs up.

A session that is still pending ends at once. The browser hanging up (call.hangUp(), or closing the tab) also ends it. Every end fires the realtime_session.ended webhook. Use GET /v1/realtime/sessions/{id} for the session’s status, end_reason, totals and one receipt per billed reply.

Limits

Budget checkpoints. Set budget_block_mc to get a wind.session.budget_checkpoint each time the session’s spend crosses another block, for example every $0.50. Add confirm_each_block: true to require approval: unless your server sends wind.session.approve_block within 30 seconds, the call ends at the next reply with budget_not_approved. Use this when your app should ask the user before a long call keeps spending.

How voice is billed

• Per reply. Each model reply is billed as its own call, with its own ledger entry: provider cost (per-modality tokens, where audio costs far more than text and reasoning tokens count as text) plus Wind’s fee plus your margin. wind.response.settled and the session’s receipts show the four numbers.
• Holds stay one reply ahead. Before each reply, Wind holds enough for the model’s worst-case reply, so replies never wait on Wind. Whatever the reply doesn’t use is released when it settles. The connection’s remaining allocation must cover that hold, which is at most about $0.19 on gpt-realtime-mini and about $0.60 on gpt-realtime-2.1.
• Replies the browser cuts off. If the user’s tab closes or their network drops mid-reply, OpenAI never reports usage for that reply. Wind charges an estimate from what it observed (the transcript, plus the session’s earlier replies). The estimate is rounded down, capped at the reply’s hold, and labelled usage_source: "estimated". Hidden reasoning isn’t charged. Failures on OpenAI’s side aren’t charged.
• Interruptions still bill unheard audio. The model generates audio faster than real time, so when the user talks over a reply, audio that was generated but never played is still billed. If your users interrupt a lot, keep max_output_tokens modest.
• Risk is judged per minute. Wind compares spend per minute with a normal conversation’s pace on that model. At 1.5× it flags the call (wind.session.spend_warning, connection.risk_warning). At 3× it pauses the connection (see Paused connections). Long calls at a normal pace are fine.

Mock mode

When the app’s provider mode is Mock, a voice session runs on Wind’s mock voice instead of a real model:

• The create response says so: mock: true, a notice, and a client with mode: "mock" and a mock_url. The control channel’s first event, wind.session.started, carries mock: true and the notice too.
• connectVoice(client) handles it for you. There’s no WebRTC: it opens the mock over a WebSocket, notices when the user has spoken (from the microphone’s level), and speaks the mock’s replies with the browser’s speech synthesis. call.mock is true, and it logs the notice to the console. The mock is turn-based, so it doesn’t listen while it’s speaking.
• Every reply is the same placeholder, which says it’s a mock and how to switch to live models. The mock doesn’t call your tools.
• Everything else is real: holds, a receipt per reply (priced from the sandbox price sheet, like mock chat), budget checkpoints, limits, ending and webhooks. That lets you test your billing and control-channel code before you go live.

Switch the app to Live models in the console to talk to a real model; nothing in your code changes.

Errors

Creating a session can fail the same ways a chat call can: 402 allocation_exhausted, 403 connection_frozen / connection_revoked / model_not_allowed, and 429 sandbox_live_cost_limited. It can also fail with 400 invalid_request (not a realtime model) and 429 too_many_sessions. The offer exchange can return 401 invalid_ticket or 502 provider_error. Full table: Errors & types.