Realtime

Introduction

Establish low-latency real-time voice sessions over WebSocket, compatible with the OpenAI Realtime API. After connecting, exchange audio and text via the event protocol—suitable for voice assistants, live translation, and similar use cases. Base URL: wss://api.gravitex.ai/v1/realtime

You must use the WebSocket protocol, not a standard HTTP POST. model must be passed as a query parameter; otherwise the gateway returns 400.

Authentication

Include the following headers when establishing the WebSocket connection:

string

required

Bearer token, e.g. Bearer sk-xxxxxxxxxx

string

required

Must be realtime=v1

Browsers cannot set custom WebSocket headers. Use OpenAI subprotocols to pass the key (equivalent to headers; parsed by the gateway):

Sec-WebSocket-Protocol: realtime, openai-insecure-api-key.sk-xxxxxxxxxx, openai-beta.realtime-v1

Connection

wss://api.gravitex.ai/v1/realtime?model=gpt-realtime-1.5

string

required

Realtime model ID, e.g. gpt-realtime-1.5, gpt-realtime-2. Must match a model available to your API key.

On success, the server returns 101 Switching Protocols. You will then receive session.created; send session.update to complete session configuration.

Session configuration

After connecting, send session.update to set instructions, output modalities, and audio format:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "You are a concise assistant. Reply in Chinese. 回答用中文回答",
    "output_modalities": ["audio"],
    "audio": {
      "input":  { "format": { "type": "audio/pcm", "rate": 24000 } },
      "output": { "format": { "type": "audio/pcm", "rate": 24000 }, "voice": "sage" }
    }
  }
}

Field	Description
`session.type`	Always `realtime`
`instructions`	System instructions controlling assistant behavior and reply language
`output_modalities`	`["audio"]` for voice output; `["text"]` for text only (no audio playback)
`audio.input.format`	Input audio format: PCM16, 24 kHz, mono
`audio.output.format`	Output audio format: same as above
`audio.output.voice`	Voice; see available values below

Available voices (when output_modalities includes audio): alloy · ash · ballad · coral · echo · sage · shimmer · verse · marin Wait for session.updated before starting the conversation.

Sending messages

Each turn has two steps:

conversation.item.create — write the user message
response.create — trigger model generation

Text input

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [
      { "type": "input_text", "text": "Hello, please introduce yourself in Chinese" }
    ]
  }
}

Audio input

Recordings must be PCM16 raw bytes (no WAV header), Base64-encoded:

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [
      { "type": "input_audio", "audio": "<base64_pcm16>" }
    ]
  }
}

You can combine multiple input types in one turn, e.g. image + text or image + audio:

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [
      { "type": "ms_image", "image": "data:image/jpeg;base64,<base64>" },
      { "type": "input_text", "text": "What's in this image?" }
    ]
  }
}

Image input limitation: Some upstream providers (e.g. Azure OpenAI Realtime) do not support images—only input_text and input_audio. Sending images may trigger a server 500 error. Vision is only available on non-Realtime Chat Completions / Responses endpoints. See Microsoft’s official note. Keep Base64 image length under 1,048,576 characters (~750 KB original).

After writing the message, trigger the response:

{ "type": "response.create" }

Receiving responses

Common server events:

Event type	Description
`response.text.delta` / `response.output_text.delta`	Text reply deltas (`output_modalities: ["text"]`)
`response.audio_transcript.delta` / `response.output_audio_transcript.delta`	Transcript deltas for audio replies
`response.audio.delta` / `response.output_audio.delta`	PCM16 audio deltas (Base64); decode before playback
`response.done`	Turn complete; includes `usage`
`error`	Error; some upstream errors close the connection—reconnect required

Audio delta fields may be audio or delta; clients should handle both.

Usage

Each response.done returns usage for that Response only, not session cumulative totals. Official docs: “The tokens used for a Response can be read from the response.done event”. For multi-turn sessions, track both per-turn and cumulative usage. Example usage structure:

{
  "total_tokens": 320,
  "input_tokens": 180,
  "output_tokens": 140,
  "input_token_details": {
    "text_tokens": 50,
    "audio_tokens": 130,
    "image_tokens": 0,
    "cached_tokens": 20,
    "cached_tokens_details": {
      "text_tokens": 8,
      "audio_tokens": 12
    }
  },
  "output_token_details": {
    "text_tokens": 40,
    "audio_tokens": 100
  }
}

Field reference

Field	Description
`total_tokens`	`input_tokens` + `output_tokens`
`input_tokens`	All input sent to the model (including conversation history)
`input_token_details.text_tokens`	Text tokens in input
`input_token_details.audio_tokens`	Audio tokens in input
`input_token_details.image_tokens`	Image tokens in input
`input_token_details.cached_tokens`	Subset of `input_tokens` (not additive)—portion that hit prompt cache; billed at cache rate
`cached_tokens_details.text_tokens`	Cached portion that is text (returned by OpenAI natively)
`cached_tokens_details.audio_tokens`	Cached portion that is audio (returned by OpenAI natively)
`output_tokens`	Tokens generated this turn
`output_token_details.text_tokens`	Text tokens in output (audio reply transcript)
`output_token_details.audio_tokens`	Audio tokens in output

output.text_tokens: Even with audio output, automatic transcript generates text tokens—this is normal billing. See OpenAI Realtime costs.
input.cached_tokens: A subset of input_tokens, not additive. OpenAI returns cached_tokens_details (text/audio split); when upstream (e.g. Azure GA) does not, the gateway estimates by text/audio input ratio.
Multi-turn cost: Each Response sends the full conversation history, so input_tokens grow over time; earlier turns often hit prompt cache with high discounts (e.g. gpt-realtime-2 audio cache $0.40/1M vs audio input$ 32/1M = 98.75% off). Official docs: “turns later in the session will be more expensive”.

Full Python example

Dependencies: pip install websockets pyaudio

import asyncio
import base64
import json
import websockets

BASE_URL = "wss://api.gravitex.ai/v1/realtime"
API_KEY  = "sk-xxxxxxxxxx"
MODEL    = "gpt-realtime-1.5"

async def main():
    url = f"{BASE_URL}?model={MODEL}"

    async with websockets.connect(url, additional_headers={
        "Authorization": f"Bearer {API_KEY}",
        "OpenAI-Beta": "realtime=v1",
    }) as ws:
        # 1. Configure session
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "type": "realtime",
                "instructions": "You are a concise assistant. Reply in Chinese.",
                "output_modalities": ["audio"],
                "audio": {
                    "input":  {"format": {"type": "audio/pcm", "rate": 24000}},
                    "output": {"format": {"type": "audio/pcm", "rate": 24000}, "voice": "sage"},
                },
            },
        }))

        # 2. Wait for session ready
        async for raw in ws:
            evt = json.loads(raw)
            if evt.get("type") in ("session.created", "session.updated"):
                break

        # 3. Send text and trigger response
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {
                "type": "message",
                "role": "user",
                "content": [{"type": "input_text", "text": "Hello"}],
            },
        }))
        await ws.send(json.dumps({"type": "response.create"}))

        # 4. Receive response
        async for raw in ws:
            evt = json.loads(raw)
            t = evt.get("type", "")
            if t in ("response.text.delta", "response.output_text.delta",
                     "response.audio_transcript.delta", "response.output_audio_transcript.delta"):
                print(evt.get("delta", ""), end="", flush=True)
            elif t in ("response.audio.delta", "response.output_audio.delta"):
                audio_b64 = evt.get("audio", "") or evt.get("delta", "")
                if audio_b64:
                    # Decode Base64 and play PCM16 audio
                    pass
            elif t == "response.done":
                print()
                resp = evt.get("response", evt)  # GA structure may differ
                usage = resp.get("usage")
                if usage:
                    print("This turn usage:", usage)
                break
            elif t == "error":
                print("Error:", evt)
                break

asyncio.run(main())

For a full interactive script (microphone recording, speaker playback, image input, per-turn and cumulative usage), see test_realtime.py.

FAQ

400 on connect: Check that the URL includes the ?model=... query parameter.
401 on connect: Invalid or unauthorized API key; call GET /v1/models to verify key and model availability.
No response after send: Confirm you received session.updated and sent response.create.
Disconnect after 500: Upstream errors close the WebSocket—reconnect.
No audio: Confirm 24 kHz PCM16 mono and correct Base64 decoding before playback.

For more event types and fields, see the OpenAI Realtime API docs.

API documentation

Chat & text

Safety & audio

Image Series

Video Series

Completions & Embeddings

Interface module

Introduction

Authentication

Connection

Session configuration

Sending messages

Text input

Audio input

Receiving responses

Usage

Field reference

Full Python example

FAQ

​Introduction

​Authentication

​Connection

​Session configuration

​Sending messages

​Text input

​Audio input

​Receiving responses

​Usage

​Field reference

​Full Python example

​FAQ

Introduction

Authentication

Connection

Session configuration

Sending messages

Text input

Audio input

Receiving responses

Usage

Field reference

Full Python example

FAQ