> ## Documentation Index > Fetch the complete documentation index at: https://docs.gravitex.ai/llms.txt > Use this file to discover all available pages before exploring further. # Realtime > WebSocket real-time voice conversation, compatible with OpenAI Realtime API ## Introduction Establish low-latency real-time voice sessions over **WebSocket**, compatible with the [OpenAI Realtime API](https://platform.openai.com/docs/guides/realtime). After connecting, exchange audio and text via the event protocol—suitable for voice assistants, live translation, and similar use cases. Base URL: `wss://api.gravitex.ai/v1/realtime` You must use the WebSocket protocol, not a standard HTTP POST. `model` must be passed as a **query parameter**; otherwise the gateway returns 400. ## Authentication Include the following headers when establishing the WebSocket connection: Bearer token, e.g. `Bearer sk-xxxxxxxxxx` Must be `realtime=v1` Browsers cannot set custom WebSocket headers. Use OpenAI subprotocols to pass the key (equivalent to headers; parsed by the gateway): ``` Sec-WebSocket-Protocol: realtime, openai-insecure-api-key.sk-xxxxxxxxxx, openai-beta.realtime-v1 ``` ## Connection ``` wss://api.gravitex.ai/v1/realtime?model=gpt-realtime-1.5 ``` Realtime model ID, e.g. `gpt-realtime-1.5`, `gpt-realtime-2`. Must match a model available to your API key. On success, the server returns **101 Switching Protocols**. You will then receive `session.created`; send `session.update` to complete session configuration. ## Session configuration After connecting, send `session.update` to set instructions, output modalities, and audio format: ```json theme={null} { "type": "session.update", "session": { "type": "realtime", "instructions": "You are a concise assistant. Reply in Chinese. 回答用中文回答", "output_modalities": ["audio"], "audio": { "input": { "format": { "type": "audio/pcm", "rate": 24000 } }, "output": { "format": { "type": "audio/pcm", "rate": 24000 }, "voice": "sage" } } } } ``` | Field | Description | | --------------------- | -------------------------------------------------------------------------- | | `session.type` | Always `realtime` | | `instructions` | System instructions controlling assistant behavior and reply language | | `output_modalities` | `["audio"]` for voice output; `["text"]` for text only (no audio playback) | | `audio.input.format` | Input audio format: **PCM16, 24 kHz, mono** | | `audio.output.format` | Output audio format: same as above | | `audio.output.voice` | Voice; see available values below | **Available voices** (when `output_modalities` includes `audio`): `alloy` · `ash` · `ballad` · `coral` · `echo` · `sage` · `shimmer` · `verse` · `marin` Wait for `session.updated` before starting the conversation. ## Sending messages Each turn has two steps: 1. `conversation.item.create` — write the user message 2. `response.create` — trigger model generation ### Text input ```json theme={null} { "type": "conversation.item.create", "item": { "type": "message", "role": "user", "content": [ { "type": "input_text", "text": "Hello, please introduce yourself in Chinese" } ] } } ``` ### Audio input Recordings must be **PCM16 raw bytes** (no WAV header), Base64-encoded: ```json theme={null} { "type": "conversation.item.create", "item": { "type": "message", "role": "user", "content": [ { "type": "input_audio", "audio": "" } ] } } ``` You can combine multiple input types in one turn, e.g. image + text or image + audio: ```json theme={null} { "type": "conversation.item.create", "item": { "type": "message", "role": "user", "content": [ { "type": "ms_image", "image": "data:image/jpeg;base64," }, { "type": "input_text", "text": "What's in this image?" } ] } } ``` **Image input limitation**: Some upstream providers (e.g. Azure OpenAI Realtime) do not support images—only `input_text` and `input_audio`. Sending images may trigger a server 500 error. Vision is only available on non-Realtime Chat Completions / Responses endpoints. See [Microsoft's official note](https://learn.microsoft.com/en-us/answers/questions/5706094/how-to-use-input-image-function-in-azure-gpt-realt). Keep Base64 image length under **1,048,576** characters (\~750 KB original). After writing the message, trigger the response: ```json theme={null} { "type": "response.create" } ``` ## Receiving responses Common server events: | Event type | Description | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------- | | `response.text.delta` / `response.output_text.delta` | Text reply deltas (`output_modalities: ["text"]`) | | `response.audio_transcript.delta` / `response.output_audio_transcript.delta` | Transcript deltas for audio replies | | `response.audio.delta` / `response.output_audio.delta` | PCM16 audio deltas (Base64); decode before playback | | `response.done` | Turn complete; includes `usage` | | `error` | Error; some upstream errors close the connection—reconnect required | Audio delta fields may be `audio` or `delta`; clients should handle both. ## Usage Each `response.done` returns **usage for that Response only**, not session cumulative totals. Official docs: *"The tokens used for a Response can be read from the response.done event"*. For multi-turn sessions, track both per-turn and cumulative usage. Example `usage` structure: ```json theme={null} { "total_tokens": 320, "input_tokens": 180, "output_tokens": 140, "input_token_details": { "text_tokens": 50, "audio_tokens": 130, "image_tokens": 0, "cached_tokens": 20, "cached_tokens_details": { "text_tokens": 8, "audio_tokens": 12 } }, "output_token_details": { "text_tokens": 40, "audio_tokens": 100 } } ``` ### Field reference | Field | Description | | ------------------------------------ | ----------------------------------------------------------------------------------------------- | | `total_tokens` | `input_tokens` + `output_tokens` | | `input_tokens` | All input sent to the model (**including conversation history**) | | `input_token_details.text_tokens` | Text tokens in input | | `input_token_details.audio_tokens` | Audio tokens in input | | `input_token_details.image_tokens` | Image tokens in input | | `input_token_details.cached_tokens` | **Subset** of `input_tokens` (not additive)—portion that hit prompt cache; billed at cache rate | | `cached_tokens_details.text_tokens` | Cached portion that is text (returned by OpenAI natively) | | `cached_tokens_details.audio_tokens` | Cached portion that is audio (returned by OpenAI natively) | | `output_tokens` | Tokens generated this turn | | `output_token_details.text_tokens` | Text tokens in output (audio reply transcript) | | `output_token_details.audio_tokens` | Audio tokens in output | 1. **output.text\_tokens**: Even with audio output, automatic transcript generates text tokens—this is normal billing. See [OpenAI Realtime costs](https://developers.openai.com/api/docs/guides/realtime-costs). 2. **input.cached\_tokens**: A subset of `input_tokens`, not additive. OpenAI returns `cached_tokens_details` (text/audio split); when upstream (e.g. Azure GA) does not, the gateway estimates by text/audio input ratio. 3. **Multi-turn cost**: Each Response sends the full conversation history, so `input_tokens` grow over time; earlier turns often hit prompt cache with high discounts (e.g. `gpt-realtime-2` audio cache $0.40/1M vs audio input $32/1M = 98.75% off). Official docs: *"turns later in the session will be more expensive"*. ## Full Python example Dependencies: `pip install websockets pyaudio` ```python theme={null} import asyncio import base64 import json import websockets BASE_URL = "wss://api.gravitex.ai/v1/realtime" API_KEY = "sk-xxxxxxxxxx" MODEL = "gpt-realtime-1.5" async def main(): url = f"{BASE_URL}?model={MODEL}" async with websockets.connect(url, additional_headers={ "Authorization": f"Bearer {API_KEY}", "OpenAI-Beta": "realtime=v1", }) as ws: # 1. Configure session await ws.send(json.dumps({ "type": "session.update", "session": { "type": "realtime", "instructions": "You are a concise assistant. Reply in Chinese.", "output_modalities": ["audio"], "audio": { "input": {"format": {"type": "audio/pcm", "rate": 24000}}, "output": {"format": {"type": "audio/pcm", "rate": 24000}, "voice": "sage"}, }, }, })) # 2. Wait for session ready async for raw in ws: evt = json.loads(raw) if evt.get("type") in ("session.created", "session.updated"): break # 3. Send text and trigger response await ws.send(json.dumps({ "type": "conversation.item.create", "item": { "type": "message", "role": "user", "content": [{"type": "input_text", "text": "Hello"}], }, })) await ws.send(json.dumps({"type": "response.create"})) # 4. Receive response async for raw in ws: evt = json.loads(raw) t = evt.get("type", "") if t in ("response.text.delta", "response.output_text.delta", "response.audio_transcript.delta", "response.output_audio_transcript.delta"): print(evt.get("delta", ""), end="", flush=True) elif t in ("response.audio.delta", "response.output_audio.delta"): audio_b64 = evt.get("audio", "") or evt.get("delta", "") if audio_b64: # Decode Base64 and play PCM16 audio pass elif t == "response.done": print() resp = evt.get("response", evt) # GA structure may differ usage = resp.get("usage") if usage: print("This turn usage:", usage) break elif t == "error": print("Error:", evt) break asyncio.run(main()) ``` For a full interactive script (microphone recording, speaker playback, image input, per-turn and cumulative usage), see `test_realtime.py`. ## FAQ * **400 on connect**: Check that the URL includes the `?model=...` query parameter. * **401 on connect**: Invalid or unauthorized API key; call `GET /v1/models` to verify key and model availability. * **No response after send**: Confirm you received `session.updated` and sent `response.create`. * **Disconnect after 500**: Upstream errors close the WebSocket—reconnect. * **No audio**: Confirm 24 kHz PCM16 mono and correct Base64 decoding before playback. For more event types and fields, see the [OpenAI Realtime API docs](https://platform.openai.com/docs/guides/realtime).