Stream images | Hume API

Stream JPEG frames from a camera or a video and receive, for each frame, the faces detected in it and their expression measurements.

Sending frames

Send each frame as one binary message containing a complete JPEG. Every frame gets exactly one reply: a measurement.result, or an error.

Configuration: send session.update before your first frame. A session.update after your first frame ends the session with config_invalid.
Rejected frames: a frame that is empty, is not a JPEG, or is over the pixel limit gets invalid_image_frame. A frame over 2MB ends the session with message_too_large.
Send rate: send frames as they are captured or played, up to 3 per second. A frame sent faster can get rate_limited and is not measured. From a live camera, drop it and send the next frame; from a recording, wait briefly and resend it. To measure a recorded video faster than it plays, send its frames to the video file endpoint.
Under load: replies slow down; a frame within the send rate is never rejected because the server is busy.
Failures: a frame whose measurement fails gets internal_error; resend it if you need its result. Five failures in a row end the session.

Matching results to frames

Replies arrive in the order you sent the frames, so pair them by position.

Keep a queue of the frames you have sent. Remove the oldest for each measurement.result, and for each error with code invalid_image_frame, rate_limited or internal_error.
Other errors do not consume a frame: they answer a JSON message you sent, or end the session.
Frames sent before session.close receive their replies before session.closed.
frame_id counts results, not frames. It does not advance for a frame that received an error, so do not use it as an index into your own frames.

Detection and measurement

The server detects the faces in each frame and lists them in faces. It measures the largest of them, up to the limit below, and lists those first, largest first.

A face is skipped, and is not listed, if its detection confidence is below face.threshold or its bounding box is under face.min_size pixels on its shorter side.
Each measured face carries its bounding box, its detection confidence, a face_id, and two sets of scores: expressions for emotional expressions such as amusement or interest, and descriptions for visible facial actions such as a smile or a jaw drop.
The faces that were not measured follow, most confident first, up to 32 faces in all. Each carries its bounding box and detection confidence, with face_id, expressions and descriptions null.
A frame with no faces is a normal result: a measurement.result with an empty faces array.
face_id tracks a face across frames in the session. A face normally keeps its ID while it is measured, and can get it back after it leaves the view or is no longer among the largest faces. IDs restart in every session and do not identify a person.

Limits

Limit	Value
Message size	2MB
Pixels per frame	8,294,400 (the area of a 3840x2160 image)
Send rate	3 frames per second
Faces measured per frame	2 by default, the largest; contact support to raise it
Faces listed per frame	32
Consecutive measurement failures	5, after which the session ends

Message flow

A session of four frames, the third a PNG sent by mistake. The rejected frame gets no frame_id, so the last result is frame_id 2. In session.closed, received.frames counts only the 3 frames accepted for measurement, and produced.measurements counts faces, so it is 4.

Direction	Message
receive	`session.created`
send	`session.update` (if you change the configuration)
receive	`session.updated` (if you sent `session.update`)
send	frame
receive	`measurement.result` (`frame_id` 0, face 0)
send	frame
receive	`measurement.result` (`frame_id` 1, faces 0 and 1)
send	frame (a PNG, not a JPEG)
receive	`error` (`invalid_image_frame`)
send	frame
receive	`measurement.result` (`frame_id` 2, face 1)
send	`session.close`
receive	`session.closed` (`received.frames` 3, `produced.measurements` 4)

Stream JPEG frames from a camera or a video and receive, for each frame, the faces detected in it and their expression measurements. ## Sending frames Send each frame as one binary message containing a complete JPEG. Every frame gets exactly one reply: a `measurement.result`, or an `error`. - **Configuration:** send `session.update` before your first frame. A `session.update` after your first frame ends the session with `config_invalid`. - **Rejected frames:** a frame that is empty, is not a JPEG, or is over the pixel limit gets `invalid_image_frame`. A frame over 2MB ends the session with `message_too_large`. - **Send rate:** send frames as they are captured or played, up to 3 per second. A frame sent faster can get `rate_limited` and is not measured. From a live camera, drop it and send the next frame; from a recording, wait briefly and resend it. To measure a recorded video faster than it plays, send its frames to the video file endpoint. - **Under load:** replies slow down; a frame within the send rate is never rejected because the server is busy. - **Failures:** a frame whose measurement fails gets `internal_error`; resend it if you need its result. Five failures in a row end the session. ## Matching results to frames Replies arrive in the order you sent the frames, so pair them by position. - Keep a queue of the frames you have sent. Remove the oldest for each `measurement.result`, and for each `error` with code `invalid_image_frame`, `rate_limited` or `internal_error`. - Other errors do not consume a frame: they answer a JSON message you sent, or end the session. - Frames sent before `session.close` receive their replies before `session.closed`. - `frame_id` counts results, not frames. It does not advance for a frame that received an `error`, so do not use it as an index into your own frames. ## Detection and measurement The server detects the faces in each frame and lists them in `faces`. It measures the largest of them, up to the limit below, and lists those first, largest first. - A face is skipped, and is not listed, if its detection confidence is below `face.threshold` or its bounding box is under `face.min_size` pixels on its shorter side. - Each measured face carries its bounding box, its detection confidence, a `face_id`, and two sets of scores: `expressions` for emotional expressions such as amusement or interest, and `descriptions` for visible facial actions such as a smile or a jaw drop. - The faces that were not measured follow, most confident first, up to 32 faces in all. Each carries its bounding box and detection confidence, with `face_id`, `expressions` and `descriptions` null. - A frame with no faces is a normal result: a `measurement.result` with an empty `faces` array. - `face_id` tracks a face across frames in the session. A face normally keeps its ID while it is measured, and can get it back after it leaves the view or is no longer among the largest faces. IDs restart in every session and do not identify a person. ## Limits | Limit | Value | |---|---| | Message size | 2MB | | Pixels per frame | 8,294,400 (the area of a 3840x2160 image) | | Send rate | 3 frames per second | | Faces measured per frame | 2 by default, the largest; contact support to raise it | | Faces listed per frame | 32 | | Consecutive measurement failures | 5, after which the session ends | ## Message flow A session of four frames, the third a PNG sent by mistake. The rejected frame gets no `frame_id`, so the last result is `frame_id` 2. In `session.closed`, `received.frames` counts only the 3 frames accepted for measurement, and `produced.measurements` counts faces, so it is 4. | Direction | Message | |---|---| | receive | `session.created` | | send | `session.update` (if you change the configuration) | | receive | `session.updated` (if you sent `session.update`) | | send | frame | | receive | `measurement.result` (`frame_id` 0, face 0) | | send | frame | | receive | `measurement.result` (`frame_id` 1, faces 0 and 1) | | send | frame (a PNG, not a JPEG) | | receive | `error` (`invalid_image_frame`) | | send | frame | | receive | `measurement.result` (`frame_id` 2, face 1) | | send | `session.close` | | receive | `session.closed` (`received.frames` 3, `produced.measurements` 4) |

Handshake

WSS

wss://api.cloud.hume.ai/v1/expression/video/realtime

Authentication

X-Hume-Api-Keystring

API Key authentication via header

Send

session.updateobjectRequired

Replaces the whole configuration; any field you omit returns to its default. Send it before your first frame, because once a frame has passed the rate check, whether or not it decoded, a `session.update` ends the session with `config_invalid`. The server confirms with `session.updated`, or ends the session with `config_invalid` if the message contains an unrecognized field or an unsupported value.

type"session.update"Required

imageobjectOptional

The format of the frames you send. Only one format is supported; any other value is rejected with config_invalid.

faceobjectOptional

The face detection settings to apply.

session.closeobjectRequired

Ends the session. The server sends any final results, then `session.closed`, and closes the connection with code 1000; on the audio endpoint, if the final measurement fails, `error` and `session.closed` with `reason` `error` follow instead. Prefer this to closing the WebSocket yourself, which ends the session immediately and discards any final results and the totals.

type"session.close"Required

image framestringRequiredformat: "binary"

A binary message containing one complete JPEG image. The server detects the faces in it and answers with one `measurement.result`, or with an `error` if the frame is rejected or its measurement fails. Frames rejected with `invalid_image_frame` or `rate_limited` are skipped and the session continues; a message over the size limit ends the session with `message_too_large`.

Receive

session.createdobjectRequired

Sent when the connection opens, before you send anything. Carries the session ID and the default configuration, which stays in effect unless you change it with session.update.

type"session.created"Required

session_idstringRequiredformat: "uuid"

Unique identifier for the session. It is also the run_id of the run the session is recorded as. Include it when contacting support about a session.

imageobjectRequired

The image format.

faceobjectRequired

The face detection settings.

session.updatedobjectRequired

Sent when a session.update is accepted. Carries the full configuration now in effect, including defaults for any fields you omitted.

type"session.updated"Required

imageobjectRequired

The image format.

faceobjectRequired

The face detection settings.

measurement.resultobjectRequired

Sent once for every frame that was measured, in the order you sent the frames, including frames in which no face was found.

type"measurement.result"Required

frame_idintegerRequired>=0

Sequential identifier of the frame within the session, starting at 0. Only frames answered with a measurement.result receive an ID, so the Nth result is for the Nth frame that was not answered with an error.

faceslist of objectsRequired

The faces detected in the frame: the measured faces first, largest first, then the faces that were not measured, in descending order of confidence.

errorobjectRequired

Sent when a message you sent is rejected or the server fails to process it. For a recoverable error the message is rejected and the session continues; for a session-ending error, `session.closed` with `reason` `error` follows. `ErrorCode` lists each code, its class, and what to do.

type"error"Required

codeenumRequired

Identifies what went wrong and whether the session continues.

Recoverable. The message is rejected and the session continues.

invalid_audio_frame: the audio frame was empty, had an odd number of bytes, or held more than 10 seconds of audio. Check how you slice audio and keep sending. Audio only.
invalid_image_frame: the frame was empty, was not a JPEG, or exceeded the pixel limit. Check the image and keep sending. Video only.
message_invalid: a text message was not valid JSON, had no type, or had an unrecognized type. Correct the message and keep sending.
rate_limited: media arrived faster than the send rate and was not processed. Audio: wait briefly and send the same frame again, so that the stream stays continuous. Video: from a live camera, drop the frame and send the next; from a recording, wait briefly and send the same frame again.

Depends on the endpoint.

internal_error: the server failed while processing media. Audio: the session ends as for a session-ending error; open a new connection. Video: the frame is skipped and the session continues; resend it if you need a result for it. Five consecutive failures end the session; frames that are rejected or have no faces to measure do not reset the count.

Session-ending. error is followed by session.closed with reason error, and the connection closes with code 1008.

config_invalid: a session.update contained an unrecognized field or an unsupported value, or arrived after you started sending media. Correct the message and open a new connection.
message_too_large: a message exceeded the size limit. Reduce the frame size and open a new connection.

messagestringRequired

A human-readable description of the error, intended for logging and debugging. The wording may change; use code for programmatic handling.

retryablebooleanRequired

false for every error that ends the session. true for every error the session survives, except internal_error on the video endpoint, where it reports whether the failure was temporary and the frame is worth resending.

session.closedobjectRequired

Sent as the last message of the session. Carries the reason the session ended and totals of what was received and produced. The connection closes immediately after it, with the close code for the reason.

type"session.closed"Required

session_idstringRequiredformat: "uuid"

Unique identifier for the session. It is also the run_id of the run the session is recorded as. Include it when contacting support about a session.

reasonenumRequired

Why the session ended. session.closed is the last message, and the WebSocket close code that follows it depends on the reason.

client_request: you sent session.close. Close code 1000.
server_shutdown: the server is restarting; reconnect at once to start a new session. Close code 1012.
idle_timeout: the server received no media frame it could use for 30 seconds; open a new session to continue. Close code 1000.
error: a session-ending error, reported in the preceding error message. Close code 1008.

Allowed values:

receivedobjectRequired

What the server received during the session.

producedobjectRequired

What the server produced during the session.

Stream JPEG frames from a camera or a video and receive, for each frame, the faces detected in it and their expression measurements.

Sending frames

Send each frame as one binary message containing a complete JPEG. Every frame gets exactly one reply: a measurement.result, or an error.

Configuration: send session.update before your first frame. A session.update after your first frame ends the session with config_invalid.
Rejected frames: a frame that is empty, is not a JPEG, or is over the pixel limit gets invalid_image_frame. A frame over 2MB ends the session with message_too_large.
Send rate: send frames as they are captured or played, up to 3 per second. A frame sent faster can get rate_limited and is not measured. From a live camera, drop it and send the next frame; from a recording, wait briefly and resend it. To measure a recorded video faster than it plays, send its frames to the video file endpoint.
Under load: replies slow down; a frame within the send rate is never rejected because the server is busy.
Failures: a frame whose measurement fails gets internal_error; resend it if you need its result. Five failures in a row end the session.

Matching results to frames

Replies arrive in the order you sent the frames, so pair them by position.

Keep a queue of the frames you have sent. Remove the oldest for each measurement.result, and for each error with code invalid_image_frame, rate_limited or internal_error.
Other errors do not consume a frame: they answer a JSON message you sent, or end the session.
Frames sent before session.close receive their replies before session.closed.
frame_id counts results, not frames. It does not advance for a frame that received an error, so do not use it as an index into your own frames.

Detection and measurement

The server detects the faces in each frame and lists them in faces. It measures the largest of them, up to the limit below, and lists those first, largest first.

A face is skipped, and is not listed, if its detection confidence is below face.threshold or its bounding box is under face.min_size pixels on its shorter side.
Each measured face carries its bounding box, its detection confidence, a face_id, and two sets of scores: expressions for emotional expressions such as amusement or interest, and descriptions for visible facial actions such as a smile or a jaw drop.
The faces that were not measured follow, most confident first, up to 32 faces in all. Each carries its bounding box and detection confidence, with face_id, expressions and descriptions null.
A frame with no faces is a normal result: a measurement.result with an empty faces array.
face_id tracks a face across frames in the session. A face normally keeps its ID while it is measured, and can get it back after it leaves the view or is no longer among the largest faces. IDs restart in every session and do not identify a person.

Limits

Limit	Value
Message size	2MB
Pixels per frame	8,294,400 (the area of a 3840x2160 image)
Send rate	3 frames per second
Faces measured per frame	2 by default, the largest; contact support to raise it
Faces listed per frame	32
Consecutive measurement failures	5, after which the session ends

Message flow

Direction	Message
receive	`session.created`
send	`session.update` (if you change the configuration)
receive	`session.updated` (if you sent `session.update`)
send	frame
receive	`measurement.result` (`frame_id` 0, face 0)
send	frame
receive	`measurement.result` (`frame_id` 1, faces 0 and 1)
send	frame (a PNG, not a JPEG)
receive	`error` (`invalid_image_frame`)
send	frame
receive	`measurement.result` (`frame_id` 2, face 1)
send	`session.close`
receive	`session.closed` (`received.frames` 3, `produced.measurements` 4)

Authentication

X-Hume-Api-Keystring

API Key authentication via header

Replaces the whole configuration; any field you omit returns to its default. Send it before your first frame, because once a frame has passed the rate check, whether or not it decoded, a session.update ends the session with config_invalid. The server confirms with session.updated, or ends the session with config_invalid if the message contains an unrecognized field or an unsupported value.

The format of the frames you send. Only one format is supported; any other value is rejected with config_invalid.

The face detection settings to apply.

Ends the session. The server sends any final results, then session.closed, and closes the connection with code 1000; on the audio endpoint, if the final measurement fails, error and session.closed with reason error follow instead. Prefer this to closing the WebSocket yourself, which ends the session immediately and discards any final results and the totals.

A binary message containing one complete JPEG image. The server detects the faces in it and answers with one measurement.result, or with an error if the frame is rejected or its measurement fails. Frames rejected with invalid_image_frame or rate_limited are skipped and the session continues; a message over the size limit ends the session with message_too_large.

Sent when the connection opens, before you send anything. Carries the session ID and the default configuration, which stays in effect unless you change it with session.update.

Unique identifier for the session. It is also the run_id of the run the session is recorded as. Include it when contacting support about a session.

The face detection settings.

Sent when a session.update is accepted. Carries the full configuration now in effect, including defaults for any fields you omitted.

The face detection settings.

Sent once for every frame that was measured, in the order you sent the frames, including frames in which no face was found.

The faces detected in the frame: the measured faces first, largest first, then the faces that were not measured, in descending order of confidence.

Sent when a message you sent is rejected or the server fails to process it. For a recoverable error the message is rejected and the session continues; for a session-ending error, session.closed with reason error follows. ErrorCode lists each code, its class, and what to do.

Identifies what went wrong and whether the session continues.

Recoverable. The message is rejected and the session continues.

invalid_audio_frame: the audio frame was empty, had an odd number of bytes, or held more than 10 seconds of audio. Check how you slice audio and keep sending. Audio only.
invalid_image_frame: the frame was empty, was not a JPEG, or exceeded the pixel limit. Check the image and keep sending. Video only.
message_invalid: a text message was not valid JSON, had no type, or had an unrecognized type. Correct the message and keep sending.
rate_limited: media arrived faster than the send rate and was not processed. Audio: wait briefly and send the same frame again, so that the stream stays continuous. Video: from a live camera, drop the frame and send the next; from a recording, wait briefly and send the same frame again.

Depends on the endpoint.

internal_error: the server failed while processing media. Audio: the session ends as for a session-ending error; open a new connection. Video: the frame is skipped and the session continues; resend it if you need a result for it. Five consecutive failures end the session; frames that are rejected or have no faces to measure do not reset the count.

Session-ending. error is followed by session.closed with reason error, and the connection closes with code 1008.

config_invalid: a session.update contained an unrecognized field or an unsupported value, or arrived after you started sending media. Correct the message and open a new connection.
message_too_large: a message exceeded the size limit. Reduce the frame size and open a new connection.

A human-readable description of the error, intended for logging and debugging. The wording may change; use code for programmatic handling.

Unique identifier for the session. It is also the run_id of the run the session is recorded as. Include it when contacting support about a session.

Why the session ended. session.closed is the last message, and the WebSocket close code that follows it depends on the reason.

client_request: you sent session.close. Close code 1000.
server_shutdown: the server is restarting; reconnect at once to start a new session. Close code 1012.
idle_timeout: the server received no media frame it could use for 30 seconds; open a new session to continue. Close code 1000.
error: a session-ending error, reported in the preceding error message. Close code 1008.

What the server received during the session.

What the server produced during the session.

Replaces the whole configuration; any field you omit returns to its default. Send it before your first frame, because once a frame has passed the rate check, whether or not it decoded, a session.update ends the session with config_invalid. The server confirms with session.updated, or ends the session with config_invalid if the message contains an unrecognized field or an unsupported value.

URL	wss://api.cloud.hume.ai/v1/expression/video/realtime
Method	GET
Status	101 Switching Protocols

Sending frames

Matching results to frames

Detection and measurement

Limits

Message flow

HandshakeTry it

Authentication

Send

Receive

Sending frames

Matching results to frames

Detection and measurement

Limits

Message flow

HandshakeTry it

Authentication

Send

Receive

Handshake

Handshake