{"asyncapi":"2.6.0","info":{"title":"Expression Measurement API","version":"1.0.0"},"channels":{"/v1/expression/audio/realtime":{"description":"Stream audio from a microphone or a file and receive expression measurements as speech happens. You do not need to detect speech or split it into segments: the server detects each stretch of speech and measures it while it is still being spoken.\n\n## Sending audio\n\nSend audio as binary messages of signed 16-bit little-endian PCM, mono, 16 kHz, with no header. Frames of around 100 ms work well. You can cut the audio at any sample boundary; the server joins the frames into one continuous stream.\n\n- **Configuration:** send `session.update` before your first audio frame. A `session.update` after your first audio frame ends the session with `config_invalid`.\n- **Rejected frames:** a frame that is empty, has an odd number of bytes, or holds more than 10 seconds of audio gets `invalid_audio_frame`, and the session continues. A message over 1MiB ends the session with `message_too_large`. Accepted frames are not acknowledged.\n- **Send rate:** stream audio in close to real time, no faster than 1 second of audio per second. A frame sent faster can get `rate_limited`, and none of its audio is measured. Wait briefly and resend the same frame, so that the stream stays continuous. To measure a recording faster than it plays, use the audio file endpoint.\n- **Failures:** if measurement fails, you receive `internal_error` and the session ends. Open a new connection.\n\n## How measurements are produced\n\nThe server detects when speech starts and stops. Each continuous stretch of speech is an **utterance**, bracketed by `utterance.start` and `utterance.end`.\n\n- **During speech:** a `measurement.result` arrives every `measurement_timer_ms` milliseconds of speech, with `trigger` set to `interval`.\n- **When speech ends:** a final measurement arrives with `trigger` set to `speech_ended`, unless the previous measurement ended less than one second earlier. Every utterance gets at least one measurement.\n- **Span:** each measurement covers the most recent 10 seconds of the utterance, or the whole utterance if it is shorter, so consecutive measurements can overlap. `audio_start_ms` and `audio_end_ms` give the span, counted from the start of the audio the server has accepted.\n- **Ending the session:** `session.close` during speech ends the utterance. You receive its final measurement, if one is due, then `utterance.end`, then `session.closed`. A session that ends with an error can leave its last utterance without an `utterance.end`.\n\n## Limits\n\n| Limit | Value |\n|---|---|\n| Audio per frame | 10 seconds (320KB) |\n| Message size | 1MiB (1,048,576 bytes) |\n| Send rate | 1 second of audio per second |\n\n## Message flow\n\nA session at the default `measurement_timer_ms` of 3000, with one utterance: speech begins 98 ms into the audio and is still in progress at 26000 ms, when `session.close` ends it.\n\n| Time in audio | Direction | Message |\n|---|---|---|\n| on connect | receive | `session.created` |\n| before audio | send | `session.update` (if you change the configuration) |\n| before audio | receive | `session.updated` (if you sent `session.update`) |\n| 0 ms onward | send | audio frames |\n| 98 ms | receive | `utterance.start` (utterance 0) |\n| 3098 ms | receive | `measurement.result` (98 to 3098 ms, `interval`) |\n| 6098 ms | receive | `measurement.result` (98 to 6098 ms, `interval`) |\n| 9098 to 24098 ms | receive | `measurement.result` every 3000 ms of speech (six more, each covering at most the most recent 10 seconds) |\n| 26000 ms | send | `session.close` |\n| 26000 ms | receive | `measurement.result` (16000 to 26000 ms, `speech_ended`) |\n| 26000 ms | receive | `utterance.end` (utterance 0) |\n| 26000 ms | receive | `session.closed` (`produced.utterances` 1, `produced.measurements` 9) |","publish":{"operationId":"subpackage_audio.Stream audio-publish","summary":"Server messages","message":{"oneOf":[{"$ref":"#/components/messages/subpackage_audio.Stream audio-server-0-audioReceiveSessionCreated"},{"$ref":"#/components/messages/subpackage_audio.Stream audio-server-1-audioReceiveSessionUpdated"},{"$ref":"#/components/messages/subpackage_audio.Stream audio-server-2-audioReceiveUtteranceStart"},{"$ref":"#/components/messages/subpackage_audio.Stream audio-server-3-audioReceiveMeasurementResult"},{"$ref":"#/components/messages/subpackage_audio.Stream audio-server-4-audioReceiveUtteranceEnd"},{"$ref":"#/components/messages/subpackage_audio.Stream audio-server-5-audioReceiveError"},{"$ref":"#/components/messages/subpackage_audio.Stream audio-server-6-audioReceiveSessionClosed"}]}},"subscribe":{"operationId":"subpackage_audio.Stream audio-subscribe","summary":"Client messages","message":{"oneOf":[{"$ref":"#/components/messages/subpackage_audio.Stream audio-client-0-audioSendSessionUpdate"},{"$ref":"#/components/messages/subpackage_audio.Stream audio-client-1-audioSendSessionClose"},{"$ref":"#/components/messages/subpackage_audio.Stream audio-client-2-audioSendAudioFrame"}]}}},"/v1/expression/video/realtime":{"description":"Stream JPEG frames from a camera or a video and receive, for each frame, the faces detected in it and their expression measurements.\n\n## Sending frames\n\nSend each frame as one binary message containing a complete JPEG. Every frame gets exactly one reply: a `measurement.result`, or an `error`.\n\n- **Configuration:** send `session.update` before your first frame. A `session.update` after your first frame ends the session with `config_invalid`.\n- **Rejected frames:** a frame that is empty, is not a JPEG, or is over the pixel limit gets `invalid_image_frame`. A frame over 2MB ends the session with `message_too_large`.\n- **Send rate:** send frames as they are captured or played, up to 3 per second. A frame sent faster can get `rate_limited` and is not measured. From a live camera, drop it and send the next frame; from a recording, wait briefly and resend it. To measure a recorded video faster than it plays, send its frames to the video file endpoint.\n- **Under load:** replies slow down; a frame within the send rate is never rejected because the server is busy.\n- **Failures:** a frame whose measurement fails gets `internal_error`; resend it if you need its result. Five failures in a row end the session.\n\n## Matching results to frames\n\nReplies arrive in the order you sent the frames, so pair them by position.\n\n- Keep a queue of the frames you have sent. Remove the oldest for each `measurement.result`, and for each `error` with code `invalid_image_frame`, `rate_limited` or `internal_error`.\n- Other errors do not consume a frame: they answer a JSON message you sent, or end the session.\n- Frames sent before `session.close` receive their replies before `session.closed`.\n- `frame_id` counts results, not frames. It does not advance for a frame that received an `error`, so do not use it as an index into your own frames.\n\n## Detection and measurement\n\nThe server detects the faces in each frame and lists them in `faces`. It measures the largest of them, up to the limit below, and lists those first, largest first.\n\n- A face is skipped, and is not listed, if its detection confidence is below `face.threshold` or its bounding box is under `face.min_size` pixels on its shorter side.\n- Each measured face carries its bounding box, its detection confidence, a `face_id`, and two sets of scores: `expressions` for emotional expressions such as amusement or interest, and `descriptions` for visible facial actions such as a smile or a jaw drop.\n- The faces that were not measured follow, most confident first, up to 32 faces in all. Each carries its bounding box and detection confidence, with `face_id`, `expressions` and `descriptions` null.\n- A frame with no faces is a normal result: a `measurement.result` with an empty `faces` array.\n- `face_id` tracks a face across frames in the session. A face normally keeps its ID while it is measured, and can get it back after it leaves the view or is no longer among the largest faces. IDs restart in every session and do not identify a person.\n\n## Limits\n\n| Limit | Value |\n|---|---|\n| Message size | 2MB |\n| Pixels per frame | 8,294,400 (the area of a 3840x2160 image) |\n| Send rate | 3 frames per second |\n| Faces measured per frame | 2 by default, the largest; contact support to raise it |\n| Faces listed per frame | 32 |\n| Consecutive measurement failures | 5, after which the session ends |\n\n## Message flow\n\nA session of four frames, the third a PNG sent by mistake. The rejected frame gets no `frame_id`, so the last result is `frame_id` 2. In `session.closed`, `received.frames` counts only the 3 frames accepted for measurement, and `produced.measurements` counts faces, so it is 4.\n\n| Direction | Message |\n|---|---|\n| receive | `session.created` |\n| send | `session.update` (if you change the configuration) |\n| receive | `session.updated` (if you sent `session.update`) |\n| send | frame |\n| receive | `measurement.result` (`frame_id` 0, face 0) |\n| send | frame |\n| receive | `measurement.result` (`frame_id` 1, faces 0 and 1) |\n| send | frame (a PNG, not a JPEG) |\n| receive | `error` (`invalid_image_frame`) |\n| send | frame |\n| receive | `measurement.result` (`frame_id` 2, face 1) |\n| send | `session.close` |\n| receive | `session.closed` (`received.frames` 3, `produced.measurements` 4) |","publish":{"operationId":"subpackage_video.Stream images-publish","summary":"Server messages","message":{"oneOf":[{"$ref":"#/components/messages/subpackage_video.Stream images-server-0-videoReceiveSessionCreated"},{"$ref":"#/components/messages/subpackage_video.Stream images-server-1-videoReceiveSessionUpdated"},{"$ref":"#/components/messages/subpackage_video.Stream images-server-2-videoReceiveMeasurementResult"},{"$ref":"#/components/messages/subpackage_video.Stream images-server-3-videoReceiveError"},{"$ref":"#/components/messages/subpackage_video.Stream images-server-4-videoReceiveSessionClosed"}]}},"subscribe":{"operationId":"subpackage_video.Stream images-subscribe","summary":"Client messages","message":{"oneOf":[{"$ref":"#/components/messages/subpackage_video.Stream images-client-0-videoSendSessionUpdate"},{"$ref":"#/components/messages/subpackage_video.Stream images-client-1-videoSendSessionClose"},{"$ref":"#/components/messages/subpackage_video.Stream images-client-2-videoSendImageFrame"}]}}}},"servers":{"Production":{"url":"wss://api.cloud.hume.ai/","protocol":"wss","x-default":true}},"components":{"messages":{"subpackage_audio.Stream audio-server-0-audioReceiveSessionCreated":{"name":"audioReceiveSessionCreated","title":"session.created","description":"Sent when the connection opens, before you send anything. Carries the session ID and the default configuration, which stays in effect unless you change it with `session.update`.","payload":{"$ref":"#/components/schemas/audio_sessionCreated"}},"subpackage_audio.Stream audio-server-1-audioReceiveSessionUpdated":{"name":"audioReceiveSessionUpdated","title":"session.updated","description":"Sent when a `session.update` is accepted. Carries the full configuration now in effect, including defaults for any fields you omitted.","payload":{"$ref":"#/components/schemas/audio_sessionUpdated"}},"subpackage_audio.Stream audio-server-2-audioReceiveUtteranceStart":{"name":"audioReceiveUtteranceStart","title":"utterance.start","description":"Sent when speech starts. Every `measurement.result` until the matching `utterance.end` belongs to this utterance.","payload":{"$ref":"#/components/schemas/audio_utteranceStart"}},"subpackage_audio.Stream audio-server-3-audioReceiveMeasurementResult":{"name":"audioReceiveMeasurementResult","title":"measurement.result","description":"Sent every `measurement_timer_ms` of speech while an utterance is in progress, and once more when the utterance ends unless the previous measurement ended less than one second earlier. Each measurement covers up to the most recent 10 seconds of the utterance.","payload":{"$ref":"#/components/schemas/audio_measurementResult"}},"subpackage_audio.Stream audio-server-4-audioReceiveUtteranceEnd":{"name":"audioReceiveUtteranceEnd","title":"utterance.end","description":"Sent when speech stops, after the utterance's final `measurement.result`, so every measurement for the utterance has arrived by the time you receive it.","payload":{"$ref":"#/components/schemas/audio_utteranceEnd"}},"subpackage_audio.Stream audio-server-5-audioReceiveError":{"name":"audioReceiveError","title":"error","description":"Sent when a message you sent is rejected or the server fails to process it. For a recoverable error the message is rejected and the session continues; for a session-ending error, `session.closed` with `reason` `error` follows. `ErrorCode` lists each code, its class, and what to do.","payload":{"$ref":"#/components/schemas/audio_error"}},"subpackage_audio.Stream audio-server-6-audioReceiveSessionClosed":{"name":"audioReceiveSessionClosed","title":"session.closed","description":"Sent as the last message of the session, after any final measurement and `utterance.end`. Carries the reason the session ended and totals of what was received and produced. The connection closes immediately after it, with the close code for the `reason`.","payload":{"$ref":"#/components/schemas/audio_sessionClosed"}},"subpackage_audio.Stream audio-client-0-audioSendSessionUpdate":{"name":"audioSendSessionUpdate","title":"session.update","description":"Replaces the whole configuration; any field you omit returns to its default. Send it before your first audio frame, because once audio has been accepted a `session.update` ends the session with `config_invalid`. The server confirms with `session.updated`, or ends the session with `config_invalid` if the message contains an unrecognized field or an unsupported value.","payload":{"$ref":"#/components/schemas/audio_sessionUpdate"}},"subpackage_audio.Stream audio-client-1-audioSendSessionClose":{"name":"audioSendSessionClose","title":"session.close","description":"Ends the session. The server sends any final results, then `session.closed`, and closes the connection with code 1000; on the audio endpoint, if the final measurement fails, `error` and `session.closed` with `reason` `error` follow instead. Prefer this to closing the WebSocket yourself, which ends the session immediately and discards any final results and the totals.","payload":{"$ref":"#/components/schemas/audio_sessionClose"}},"subpackage_audio.Stream audio-client-2-audioSendAudioFrame":{"name":"audioSendAudioFrame","title":"audio frame","description":"A binary message of signed 16-bit little-endian PCM audio, mono, 16 kHz, with no header. The server appends it to the session's audio and measures speech as it is detected; accepted frames are not acknowledged. A frame that is empty, does not contain whole samples, or exceeds the frame limit is rejected with `invalid_audio_frame`, and a frame that arrives faster than the send rate can be rejected with `rate_limited`; in both cases the session continues.","payload":{"$ref":"#/components/schemas/audio_audioFrame"}},"subpackage_video.Stream images-server-0-videoReceiveSessionCreated":{"name":"videoReceiveSessionCreated","title":"session.created","description":"Sent when the connection opens, before you send anything. Carries the session ID and the default configuration, which stays in effect unless you change it with `session.update`.","payload":{"$ref":"#/components/schemas/video_sessionCreated"}},"subpackage_video.Stream images-server-1-videoReceiveSessionUpdated":{"name":"videoReceiveSessionUpdated","title":"session.updated","description":"Sent when a `session.update` is accepted. Carries the full configuration now in effect, including defaults for any fields you omitted.","payload":{"$ref":"#/components/schemas/video_sessionUpdated"}},"subpackage_video.Stream images-server-2-videoReceiveMeasurementResult":{"name":"videoReceiveMeasurementResult","title":"measurement.result","description":"Sent once for every frame that was measured, in the order you sent the frames, including frames in which no face was found.","payload":{"$ref":"#/components/schemas/video_measurementResult"}},"subpackage_video.Stream images-server-3-videoReceiveError":{"name":"videoReceiveError","title":"error","description":"Sent when a message you sent is rejected or the server fails to process it. For a recoverable error the message is rejected and the session continues; for a session-ending error, `session.closed` with `reason` `error` follows. `ErrorCode` lists each code, its class, and what to do.","payload":{"$ref":"#/components/schemas/video_error"}},"subpackage_video.Stream images-server-4-videoReceiveSessionClosed":{"name":"videoReceiveSessionClosed","title":"session.closed","description":"Sent as the last message of the session. Carries the reason the session ended and totals of what was received and produced. The connection closes immediately after it, with the close code for the `reason`.","payload":{"$ref":"#/components/schemas/video_sessionClosed"}},"subpackage_video.Stream images-client-0-videoSendSessionUpdate":{"name":"videoSendSessionUpdate","title":"session.update","description":"Replaces the whole configuration; any field you omit returns to its default. Send it before your first frame, because once a frame has passed the rate check, whether or not it decoded, a `session.update` ends the session with `config_invalid`. The server confirms with `session.updated`, or ends the session with `config_invalid` if the message contains an unrecognized field or an unsupported value.","payload":{"$ref":"#/components/schemas/video_sessionUpdate"}},"subpackage_video.Stream images-client-1-videoSendSessionClose":{"name":"videoSendSessionClose","title":"session.close","description":"Ends the session. The server sends any final results, then `session.closed`, and closes the connection with code 1000; on the audio endpoint, if the final measurement fails, `error` and `session.closed` with `reason` `error` follow instead. Prefer this to closing the WebSocket yourself, which ends the session immediately and discards any final results and the totals.","payload":{"$ref":"#/components/schemas/video_sessionClose"}},"subpackage_video.Stream images-client-2-videoSendImageFrame":{"name":"videoSendImageFrame","title":"image frame","description":"A binary message containing one complete JPEG image. The server detects the faces in it and answers with one `measurement.result`, or with an `error` if the frame is rejected or its measurement fails. Frames rejected with `invalid_image_frame` or `rate_limited` are skipped and the session continues; a message over the size limit ends the session with `message_too_large`.","payload":{"$ref":"#/components/schemas/video_imageFrame"}}},"schemas":{"SessionId":{"type":"string","format":"uuid","description":"Unique identifier for the session. It is also the `run_id` of the run the session is recorded as. Include it when contacting support about a session.","title":"SessionId"},"AudioConfig":{"type":"object","properties":{"type":{"type":"string","enum":["audio/pcm"]},"encoding":{"type":"string","enum":["s16le"],"description":"The sample encoding. `s16le` is signed 16-bit little-endian PCM."},"sample_rate":{"type":"integer","description":"The sample rate in Hz."},"channels":{"type":"integer"}},"required":["type","encoding","sample_rate","channels"],"description":"The audio format.","title":"AudioConfig"},"MeasurementTimerMs":{"type":"integer","minimum":3000,"maximum":10000,"description":"The interval between measurements while speech is in progress, in milliseconds of speech.","title":"MeasurementTimerMs"},"audio_sessionCreated":{"type":"object","properties":{"type":{"type":"string","enum":["session.created"]},"session_id":{"$ref":"#/components/schemas/SessionId"},"audio":{"$ref":"#/components/schemas/AudioConfig"},"measurement_timer_ms":{"$ref":"#/components/schemas/MeasurementTimerMs"}},"required":["type","session_id","audio","measurement_timer_ms"],"description":"The session ID and the configuration in effect when the connection opened.","title":"audio_sessionCreated"},"audio_sessionUpdated":{"type":"object","properties":{"type":{"type":"string","enum":["session.updated"]},"audio":{"$ref":"#/components/schemas/AudioConfig"},"measurement_timer_ms":{"$ref":"#/components/schemas/MeasurementTimerMs"}},"required":["type","audio","measurement_timer_ms"],"description":"The full configuration in effect after a `session.update`, including defaults for any fields you omitted.","title":"audio_sessionUpdated"},"audio_utteranceStart":{"type":"object","properties":{"type":{"type":"string","enum":["utterance.start"]},"utterance_id":{"type":"integer","minimum":0,"description":"Sequential identifier of the utterance within the session, starting at 0."},"audio_start_ms":{"type":"integer","minimum":0,"description":"Where speech started, in milliseconds from the start of the session's audio."}},"required":["type","utterance_id","audio_start_ms"],"description":"The start of an utterance. Every `measurement.result` until the matching `utterance.end` belongs to it.","title":"audio_utteranceStart"},"MeasurementTrigger":{"type":"string","enum":["interval","speech_ended"],"description":"Why the measurement was produced.\n\n- `interval`: the measurement interval (`measurement_timer_ms`) elapsed while speech continued.\n- `speech_ended`: the utterance ended, or the audio ended while speech was in progress.\n\nA final measurement is skipped when the previous one ended less than one second earlier, so the last measurement of an utterance may carry the trigger `interval`. Where an utterance ended is reported separately: as an `utterance.end` message in a realtime session, and in `utterances` in an audio file response.","title":"MeasurementTrigger"},"AudioExpressionName":{"type":"string","description":"The name of an expression that can be detected in speech; see the [full list of audio expressions](/expression-measurement/docs/audio/measurements#expressions). Names are lowercase and may contain spaces and punctuation. New names may be added over time; treat a name you do not recognize as a new prediction.","title":"AudioExpressionName"},"AudioExpressionScore":{"type":"object","properties":{"name":{"$ref":"#/components/schemas/AudioExpressionName"},"probability":{"type":"number","format":"double","minimum":0,"maximum":1,"description":"The probability that this expression is present, rounded to four decimal places."}},"required":["name","probability"],"description":"A scored expression detected in the speech.","title":"AudioExpressionScore"},"VoiceAttributeName":{"type":"string","description":"The name of a quality of the voice or the audio, such as a manner of speaking, a vocal timbre or a background sound; see the [full list of voice attributes](/expression-measurement/docs/audio/measurements#voice-attributes). Names are lowercase and may contain spaces and punctuation. New names may be added over time; treat a name you do not recognize as a new prediction.","title":"VoiceAttributeName"},"VoiceAttributeScore":{"type":"object","properties":{"name":{"$ref":"#/components/schemas/VoiceAttributeName"},"probability":{"type":"number","format":"double","minimum":0,"maximum":1,"description":"The probability that this quality is present, rounded to four decimal places."}},"required":["name","probability"],"description":"A scored quality of the voice or the audio.","title":"VoiceAttributeScore"},"audio_measurementResult":{"type":"object","properties":{"type":{"type":"string","enum":["measurement.result"]},"measurement_id":{"type":"integer","minimum":0,"description":"Sequential identifier of the measurement within the session, starting at 0."},"utterance_id":{"type":"integer","minimum":0,"description":"The utterance this measurement belongs to, as given in `utterance.start`."},"audio_start_ms":{"type":"integer","minimum":0,"description":"Start of the window measured, in milliseconds from the start of the session's audio. Never earlier than the start of the utterance."},"audio_end_ms":{"type":"integer","minimum":0,"description":"End of the window measured, in milliseconds from the start of the session's audio."},"trigger":{"$ref":"#/components/schemas/MeasurementTrigger"},"expressions":{"type":"array","items":{"$ref":"#/components/schemas/AudioExpressionScore"},"description":"Expressions detected in the speech, such as `joy`, `anger` and `chuckling`, in descending order of probability. Scores with equal probability are ordered by the model's underlying score, then by name. Only expressions judged present are included, each against a threshold set for that expression, so the array may be empty. `probability` is the measured chance that the expression is truly present."},"voice_attributes":{"type":"array","items":{"$ref":"#/components/schemas/VoiceAttributeScore"},"description":"Qualities of the voice and the audio, such as `husky`, `fast` and `background music`, in descending order of probability. Scores with equal probability are ordered by the model's underlying score, then by name. Only scores of 0.725 or above are included, so the array may be empty. `probability` is calibrated: the expected intensity a human rater would give the quality."}},"required":["type","measurement_id","utterance_id","audio_start_ms","audio_end_ms","trigger","expressions","voice_attributes"],"description":"Expression measurements for a window of an utterance. The window spans `audio_start_ms` to `audio_end_ms` and covers up to the most recent 10 seconds of the utterance. An interval measurement taken as speech stops can end shortly after the utterance does. Consecutive measurements of an utterance overlap, except interval measurements at a `measurement_timer_ms` of 10000, which meet end to end.","title":"audio_measurementResult"},"audio_utteranceEnd":{"type":"object","properties":{"type":{"type":"string","enum":["utterance.end"]},"utterance_id":{"type":"integer","minimum":0,"description":"The utterance that ended, as given in `utterance.start`."},"audio_start_ms":{"type":"integer","minimum":0,"description":"Where the utterance started, in milliseconds from the start of the session's audio. Matches the `utterance.start` message."},"audio_end_ms":{"type":"integer","minimum":0,"description":"Where the utterance ended, in milliseconds from the start of the session's audio."}},"required":["type","utterance_id","audio_start_ms","audio_end_ms"],"description":"The end of an utterance, with its full extent in the audio.","title":"audio_utteranceEnd"},"ErrorCode":{"type":"string","enum":["invalid_audio_frame","invalid_image_frame","message_invalid","rate_limited","config_invalid","message_too_large","internal_error"],"description":"Identifies what went wrong and whether the session continues.\n\n**Recoverable.** The message is rejected and the session continues.\n\n- `invalid_audio_frame`: the audio frame was empty, had an odd number of bytes, or held more than 10 seconds of audio. Check how you slice audio and keep sending. Audio only.\n- `invalid_image_frame`: the frame was empty, was not a JPEG, or exceeded the pixel limit. Check the image and keep sending. Video only.\n- `message_invalid`: a text message was not valid JSON, had no `type`, or had an unrecognized `type`. Correct the message and keep sending.\n- `rate_limited`: media arrived faster than the send rate and was not processed. Audio: wait briefly and send the same frame again, so that the stream stays continuous. Video: from a live camera, drop the frame and send the next; from a recording, wait briefly and send the same frame again.\n\n**Depends on the endpoint.**\n\n- `internal_error`: the server failed while processing media. Audio: the session ends as for a session-ending error; open a new connection. Video: the frame is skipped and the session continues; resend it if you need a result for it. Five consecutive failures end the session; frames that are rejected or have no faces to measure do not reset the count.\n\n**Session-ending.** `error` is followed by `session.closed` with `reason` `error`, and the connection closes with code 1008.\n\n- `config_invalid`: a `session.update` contained an unrecognized field or an unsupported value, or arrived after you started sending media. Correct the message and open a new connection.\n- `message_too_large`: a message exceeded the size limit. Reduce the frame size and open a new connection.","title":"ErrorCode"},"audio_error":{"type":"object","properties":{"type":{"type":"string","enum":["error"]},"code":{"$ref":"#/components/schemas/ErrorCode"},"message":{"type":"string","description":"A human-readable description of the error, intended for logging and debugging. The wording may change; use `code` for programmatic handling."},"retryable":{"type":"boolean","description":"`false` for every error that ends the session. `true` for every error the session survives, except `internal_error` on the video endpoint, where it reports whether the failure was temporary and the frame is worth resending."}},"required":["type","code","message","retryable"],"description":"A rejected message or a processing failure. `code` says what happened and whether the session continues.","title":"audio_error"},"CloseReason":{"type":"string","enum":["client_request","server_shutdown","idle_timeout","error"],"description":"Why the session ended. `session.closed` is the last message, and the WebSocket close code that follows it depends on the reason.\n\n- `client_request`: you sent `session.close`. Close code 1000.\n- `server_shutdown`: the server is restarting; reconnect at once to start a new session. Close code 1012.\n- `idle_timeout`: the server received no media frame it could use for 30 seconds; open a new session to continue. Close code 1000.\n- `error`: a session-ending error, reported in the preceding `error` message. Close code 1008.","title":"CloseReason"},"AudioReceived":{"type":"object","properties":{"audio_duration_ms":{"type":"integer","minimum":0,"description":"Total duration of audio accepted, in milliseconds, whether or not it contained speech."},"frames":{"type":"integer","minimum":0,"description":"Number of audio frames accepted. Frames rejected with an error are not counted, so comparing this with the number you sent tells you how many were rejected."}},"required":["audio_duration_ms","frames"],"description":"What the server received during the session.","title":"AudioReceived"},"AudioProduced":{"type":"object","properties":{"utterances":{"type":"integer","minimum":0,"description":"Number of utterances detected."},"measurements":{"type":"integer","minimum":0,"description":"Number of `measurement.result` messages sent. This counts messages; the video endpoint's field of the same name counts faces."}},"required":["utterances","measurements"],"description":"What the server produced during the session.","title":"AudioProduced"},"audio_sessionClosed":{"type":"object","properties":{"type":{"type":"string","enum":["session.closed"]},"session_id":{"$ref":"#/components/schemas/SessionId"},"reason":{"$ref":"#/components/schemas/CloseReason"},"received":{"$ref":"#/components/schemas/AudioReceived"},"produced":{"$ref":"#/components/schemas/AudioProduced"}},"required":["type","session_id","reason","received","produced"],"description":"Why the session ended, and totals of what was received and produced.","title":"audio_sessionClosed"},"AudioConfigUpdate":{"type":"object","properties":{"type":{"type":"string","enum":["audio/pcm"],"default":"audio/pcm"},"encoding":{"type":"string","enum":["s16le"],"default":"s16le","description":"The sample encoding. `s16le` is signed 16-bit little-endian PCM."},"sample_rate":{"type":"integer","default":16000,"description":"The sample rate in Hz."},"channels":{"type":"integer","default":1}},"description":"The format of the audio you send. Only one format is supported; any other value is rejected with `config_invalid`.","title":"AudioConfigUpdate"},"audio_sessionUpdate":{"type":"object","properties":{"type":{"type":"string","enum":["session.update"]},"audio":{"$ref":"#/components/schemas/AudioConfigUpdate"},"measurement_timer_ms":{"type":"integer","minimum":3000,"maximum":10000,"default":3000,"description":"The interval between measurements while speech is in progress, in milliseconds of speech. Lower values produce measurements more often; at the maximum, interval measurements are back to back rather than overlapping."}},"required":["type"],"description":"The configuration to apply to the session. Omitted fields are reset to their defaults.","title":"audio_sessionUpdate"},"audio_sessionClose":{"type":"object","properties":{"type":{"type":"string","enum":["session.close"]}},"required":["type"],"description":"A request to end the session.","title":"audio_sessionClose"},"audio_audioFrame":{"type":"string","format":"binary","description":"Raw audio: signed 16-bit little-endian PCM, mono, 16 kHz, with no header. A frame must be non-empty, contain whole samples, and stay within the frame limit given in the channel description. Frame boundaries do not need to align with anything.","title":"audio_audioFrame"},"ImageConfig":{"type":"object","properties":{"type":{"type":"string","enum":["image/jpeg"]}},"required":["type"],"description":"The image format.","title":"ImageConfig"},"FaceConfig":{"type":"object","properties":{"threshold":{"type":"number","format":"double","minimum":0,"maximum":1,"description":"Minimum detection confidence for a face to be measured. Lower values include more faces at the risk of false detections. This scale is specific to this API; thresholds tuned for other face detection APIs do not carry over. Reported rounded to four decimal places."},"min_size":{"type":"integer","minimum":1,"maximum":4294967295,"description":"Minimum face size to measure, as the shorter side of the face's bounding box in pixels of the submitted image."}},"required":["threshold","min_size"],"description":"The face detection settings.","title":"FaceConfig"},"video_sessionCreated":{"type":"object","properties":{"type":{"type":"string","enum":["session.created"]},"session_id":{"$ref":"#/components/schemas/SessionId"},"image":{"$ref":"#/components/schemas/ImageConfig"},"face":{"$ref":"#/components/schemas/FaceConfig"}},"required":["type","session_id","image","face"],"description":"The session ID and the configuration in effect when the connection opened.","title":"video_sessionCreated"},"video_sessionUpdated":{"type":"object","properties":{"type":{"type":"string","enum":["session.updated"]},"image":{"$ref":"#/components/schemas/ImageConfig"},"face":{"$ref":"#/components/schemas/FaceConfig"}},"required":["type","image","face"],"description":"The full configuration in effect after a `session.update`, including defaults for any fields you omitted.","title":"video_sessionUpdated"},"BoundingBox":{"type":"array","items":{"type":"number","format":"double"},"description":"The location of the face in the submitted image, as `[x0, y0, x1, y1]`: the top-left and bottom-right corners in pixels, with the origin at the top-left of the image. Coordinates are in the pixels as stored in the file: EXIF orientation is not applied.","title":"BoundingBox"},"VideoExpressionName":{"type":"string","description":"The name of an expression that can be detected on a face, from a vocabulary that differs from that of audio expressions; see the [full list of video expressions](/expression-measurement/docs/video/measurements#expressions). Names are lowercase and may contain spaces and punctuation. New names may be added over time; treat a name you do not recognize as a new prediction.","title":"VideoExpressionName"},"VideoExpressionScore":{"type":"object","properties":{"name":{"$ref":"#/components/schemas/VideoExpressionName"},"probability":{"type":"number","format":"double","minimum":0,"maximum":1,"description":"The probability that this expression is present, rounded to four decimal places."}},"required":["name","probability"],"description":"A scored expression detected on the face.","title":"VideoExpressionScore"},"VideoDescriptionName":{"type":"string","description":"The name of a visible facial action; see the [full list of video descriptions](/expression-measurement/docs/video/measurements#descriptions). Names are lowercase and may contain spaces and punctuation. New names may be added over time; treat a name you do not recognize as a new prediction.","title":"VideoDescriptionName"},"VideoDescriptionScore":{"type":"object","properties":{"name":{"$ref":"#/components/schemas/VideoDescriptionName"},"probability":{"type":"number","format":"double","minimum":0,"maximum":1,"description":"The probability that this action is present, rounded to four decimal places."}},"required":["name","probability"],"description":"A scored facial action.","title":"VideoDescriptionScore"},"Face":{"type":"object","properties":{"face_id":{"type":["integer","null"],"minimum":0,"description":"Tracks this face across the images of a realtime session or a file request. IDs are assigned from 0 in the order faces are first measured, largest first within an image. A face normally keeps its ID while it is measured, and can get it back after it leaves the view or is no longer among the largest faces. The server remembers up to 32 faces, so a face that returns after many others have been measured may receive a new ID. IDs restart in every session and every file request, unless a file request continues them with a `face_state`, and do not identify a person. Null for a face that was not measured."},"bbox":{"$ref":"#/components/schemas/BoundingBox"},"confidence":{"type":"number","format":"double","minimum":0,"maximum":1,"description":"Confidence that the bounding box contains a face. Always at or above the configured `threshold`. Unrelated to the expression scores."},"expressions":{"type":["array","null"],"items":{"$ref":"#/components/schemas/VideoExpressionScore"},"description":"Expressions detected on the face, such as `amusement`, `interest` and `surprise (positive)`, in descending order of probability, then by name. Only scores above 0.1 are included, so the array may be empty. `probability` is the chance a human rater would name the expression for this face, independently for each expression. Null for a face that was not measured."},"descriptions":{"type":["array","null"],"items":{"$ref":"#/components/schemas/VideoDescriptionScore"},"description":"Visible facial actions, such as `smile`, `jaw drop` and `wide-eyed`, in descending order of probability, then by name. Only scores above 0.1 are included, so the array may be empty. Null for a face that was not measured."}},"required":["face_id","bbox","confidence","expressions","descriptions"],"description":"A face detected in the image, with its expression measurements if it was measured. Only the largest faces in an image are measured; the others carry their bounding box and detection confidence, with `face_id`, `expressions` and `descriptions` null.","title":"Face"},"video_measurementResult":{"type":"object","properties":{"type":{"type":"string","enum":["measurement.result"]},"frame_id":{"type":"integer","minimum":0,"description":"Sequential identifier of the frame within the session, starting at 0. Only frames answered with a `measurement.result` receive an ID, so the Nth result is for the Nth frame that was not answered with an `error`."},"faces":{"type":"array","items":{"$ref":"#/components/schemas/Face"},"description":"The faces detected in the frame: the measured faces first, largest first, then the faces that were not measured, in descending order of confidence."}},"required":["type","frame_id","faces"],"description":"Expression measurements for every face in one frame. `faces` is empty when no face was detected; this is a normal result, not an error.","title":"video_measurementResult"},"video_error":{"type":"object","properties":{"type":{"type":"string","enum":["error"]},"code":{"$ref":"#/components/schemas/ErrorCode"},"message":{"type":"string","description":"A human-readable description of the error, intended for logging and debugging. The wording may change; use `code` for programmatic handling."},"retryable":{"type":"boolean","description":"`false` for every error that ends the session. `true` for every error the session survives, except `internal_error` on the video endpoint, where it reports whether the failure was temporary and the frame is worth resending."}},"required":["type","code","message","retryable"],"description":"A rejected message or a processing failure. `code` says what happened and whether the session continues.","title":"video_error"},"VideoReceived":{"type":"object","properties":{"frames":{"type":"integer","minimum":0,"description":"Number of frames accepted for measurement. Frames rejected before detection are not counted, so comparing this with the number you sent tells you how many were rejected. A frame whose measurement then failed with `internal_error` is counted although it produced no result, so this can exceed the number of results you received."}},"required":["frames"],"description":"What the server received during the session.","title":"VideoReceived"},"VideoProduced":{"type":"object","properties":{"measurements":{"type":"integer","minimum":0,"description":"Number of faces measured across all frames: a frame with two measured faces counts as two, and a frame with none counts as zero. A face listed without measurements is not counted. This counts faces; the audio endpoint's field of the same name counts messages."}},"required":["measurements"],"description":"What the server produced during the session.","title":"VideoProduced"},"video_sessionClosed":{"type":"object","properties":{"type":{"type":"string","enum":["session.closed"]},"session_id":{"$ref":"#/components/schemas/SessionId"},"reason":{"$ref":"#/components/schemas/CloseReason"},"received":{"$ref":"#/components/schemas/VideoReceived"},"produced":{"$ref":"#/components/schemas/VideoProduced"}},"required":["type","session_id","reason","received","produced"],"description":"Why the session ended, and totals of what was received and produced.","title":"video_sessionClosed"},"ImageConfigUpdate":{"type":"object","properties":{"type":{"type":"string","enum":["image/jpeg"],"default":"image/jpeg"}},"description":"The format of the frames you send. Only one format is supported; any other value is rejected with `config_invalid`.","title":"ImageConfigUpdate"},"FaceConfigUpdate":{"type":"object","properties":{"threshold":{"type":"number","format":"double","minimum":0,"maximum":1,"default":0.9,"description":"Minimum detection confidence for a face to be measured. Lower values include more faces at the risk of false detections. This scale is specific to this API; thresholds tuned for other face detection APIs do not carry over."},"min_size":{"type":"integer","minimum":1,"maximum":4294967295,"default":60,"description":"Minimum face size to measure, as the shorter side of the face's bounding box in pixels of the submitted image."}},"description":"The face detection settings to apply.","title":"FaceConfigUpdate"},"video_sessionUpdate":{"type":"object","properties":{"type":{"type":"string","enum":["session.update"]},"image":{"$ref":"#/components/schemas/ImageConfigUpdate"},"face":{"$ref":"#/components/schemas/FaceConfigUpdate"}},"required":["type"],"description":"The configuration to apply to the session. Omitted fields are reset to their defaults.","title":"video_sessionUpdate"},"video_sessionClose":{"type":"object","properties":{"type":{"type":"string","enum":["session.close"]}},"required":["type"],"description":"A request to end the session.","title":"video_sessionClose"},"video_imageFrame":{"type":"string","format":"binary","description":"One complete JPEG image, within the size and pixel limits given in the channel description.","title":"video_imageFrame"}}}}