Sessions

A session is one WebSocket connection to a realtime endpoint. It starts when the server sends session.created and ends when the connection closes, normally right after session.closed. The audio and video endpoints share this protocol. Audio realtime and Video realtime cover each endpoint’s media format, configuration, and results.

Messages

Text messages are JSON objects identified by a type field. Media travels as binary frames. Messages follow the compatibility rules.

The audio endpoint also sends utterance.start and utterance.end to mark the start and end of each stretch of speech.

Lifecycle

Configuration

The example below changes one field. The confirmation shows the full configuration that results.

{
  "type": "session.update",
  "measurement_timer_ms": 5000
}

{
  "type": "session.updated",
  "audio": {
    "type": "audio/pcm",
    "encoding": "s16le",
    "sample_rate": 16000,
    "channels": 1
  },
  "measurement_timer_ms": 5000
}

1. Omitted fields return to their defaults. session.update replaces the configuration rather than merging into it, so include every field you want in effect.
2. The configuration locks at the first media frame the server takes. A later session.update is rejected with config_invalid, and the session ends. Which frames lock it differs by endpoint; see Audio configuration and Video configuration.
3. Unrecognized fields and unsupported values end the session with config_invalid.
4. Several updates may be sent until the configuration locks. Each replaces the configuration and is confirmed with session.updated.

The fields each endpoint accepts are listed under Audio configuration and Video configuration.

Closing

To end a session, send session.close and keep reading until session.closed arrives. Results still in progress are delivered before it. On the audio endpoint, if the final measurement fails, an error arrives first and session.closed carries reason set to error.

{
  "type": "session.close"
}

session.closed states the reason the session ended and totals what the server received and produced. The totals differ by endpoint; see the audio and video session summaries.

{
  "type": "session.closed",
  "session_id": "01926f3a-5b1c-7d2e-8f40-3a9b7c1d2e5f",
  "reason": "client_request",
  "received": {
    "audio_duration_ms": 26000,
    "frames": 260
  },
  "produced": {
    "utterances": 1,
    "measurements": 9
  }
}

After session.closed, the server closes the WebSocket with the code listed for the reason. Close codes describes every code the server closes with.

If the server cannot continue a session, it closes the connection with code 1011 without sending session.closed, so no reason is given. See Connection failures.

If the client closes the WebSocket instead of sending session.close, the session ends at once. The server sends nothing further, so results in progress and session.closed are lost.

Connection behavior

1. Messages arrive in order. The server sends messages in the order it produces them, and results follow the order in which media was accepted.
2. The server never sends pings. It answers any ping from the client with a pong.
3. A session closes after 30 seconds without a media frame the server can use. It ends with session.closed and reason set to idle_timeout. Accepted frames reset the timer, as do audio frames rejected with rate_limited. Invalid frames, video frames rejected with rate_limited, text messages, and pings do not. To pause for longer, end the session and open a new one when you resume.
4. A session cannot be resumed. However it ended, open a new connection to start another.

Time and identifiers

Time values are integers in milliseconds, measured from the start of the session’s media. session_id is a UUID. The other identifiers (measurement_id, utterance_id, frame_id, and face_id) are integers that start at 0 in every session, increase as the session progresses, and mean nothing outside it. face_id is null for a face that was not measured.

After a session

Every session is recorded as a run, and its session_id is the run’s ID. The run endpoints return its status, the configuration it ran under, and the problems it reported. They do not return measurements, so keep the results you need from the session itself. See Runs.

Next steps