Stream images
Stream JPEG frames from a camera or a video and receive, for each frame, the faces detected in it and their expression measurements.
Sending frames
Send each frame as one binary message containing a complete JPEG. Every frame gets exactly one reply: a measurement.result, or an error.
- Configuration: send
session.updatebefore your first frame. Asession.updateafter your first frame ends the session withconfig_invalid. - Rejected frames: a frame that is empty, is not a JPEG, or is over the pixel limit gets
invalid_image_frame. A frame over 2MB ends the session withmessage_too_large. - Send rate: send frames as they are captured or played, up to 3 per second. A frame sent faster can get
rate_limitedand is not measured. From a live camera, drop it and send the next frame; from a recording, wait briefly and resend it. To measure a recorded video faster than it plays, send its frames to the video file endpoint. - Under load: replies slow down; a frame within the send rate is never rejected because the server is busy.
- Failures: a frame whose measurement fails gets
internal_error; resend it if you need its result. Five failures in a row end the session.
Matching results to frames
Replies arrive in the order you sent the frames, so pair them by position.
- Keep a queue of the frames you have sent. Remove the oldest for each
measurement.result, and for eacherrorwith codeinvalid_image_frame,rate_limitedorinternal_error. - Other errors do not consume a frame: they answer a JSON message you sent, or end the session.
- Frames sent before
session.closereceive their replies beforesession.closed. frame_idcounts results, not frames. It does not advance for a frame that received anerror, so do not use it as an index into your own frames.
Detection and measurement
The server detects the faces in each frame and lists them in faces. It measures the largest of them, up to the limit below, and lists those first, largest first.
- A face is skipped, and is not listed, if its detection confidence is below
face.thresholdor its bounding box is underface.min_sizepixels on its shorter side. - Each measured face carries its bounding box, its detection confidence, a
face_id, and two sets of scores:expressionsfor emotional expressions such as amusement or interest, anddescriptionsfor visible facial actions such as a smile or a jaw drop. - The faces that were not measured follow, most confident first, up to 32 faces in all. Each carries its bounding box and detection confidence, with
face_id,expressionsanddescriptionsnull. - A frame with no faces is a normal result: a
measurement.resultwith an emptyfacesarray. face_idtracks a face across frames in the session. A face normally keeps its ID while it is measured, and can get it back after it leaves the view or is no longer among the largest faces. IDs restart in every session and do not identify a person.
Limits
Message flow
A session of four frames, the third a PNG sent by mistake. The rejected frame gets no frame_id, so the last result is frame_id 2. In session.closed, received.frames counts only the 3 frames accepted for measurement, and produced.measurements counts faces, so it is 4.
Handshake
Authentication
Send
Replaces the whole configuration; any field you omit returns to its default. Send it before your first frame, because once a frame has passed the rate check, whether or not it decoded, a session.update ends the session with config_invalid. The server confirms with session.updated, or ends the session with config_invalid if the message contains an unrecognized field or an unsupported value.
The format of the frames you send. Only one format is supported; any other value is rejected with config_invalid.
Ends the session. The server sends any final results, then session.closed, and closes the connection with code 1000; on the audio endpoint, if the final measurement fails, error and session.closed with reason error follow instead. Prefer this to closing the WebSocket yourself, which ends the session immediately and discards any final results and the totals.
A binary message containing one complete JPEG image. The server detects the faces in it and answers with one measurement.result, or with an error if the frame is rejected or its measurement fails. Frames rejected with invalid_image_frame or rate_limited are skipped and the session continues; a message over the size limit ends the session with message_too_large.
Receive
Sent when the connection opens, before you send anything. Carries the session ID and the default configuration, which stays in effect unless you change it with session.update.
Unique identifier for the session. It is also the run_id of the run the session is recorded as. Include it when contacting support about a session.
Sent when a session.update is accepted. Carries the full configuration now in effect, including defaults for any fields you omitted.
Sequential identifier of the frame within the session, starting at 0. Only frames answered with a measurement.result receive an ID, so the Nth result is for the Nth frame that was not answered with an error.
The faces detected in the frame: the measured faces first, largest first, then the faces that were not measured, in descending order of confidence.
Sent when a message you sent is rejected or the server fails to process it. For a recoverable error the message is rejected and the session continues; for a session-ending error, session.closed with reason error follows. ErrorCode lists each code, its class, and what to do.
Identifies what went wrong and whether the session continues.
Recoverable. The message is rejected and the session continues.
invalid_audio_frame: the audio frame was empty, had an odd number of bytes, or held more than 10 seconds of audio. Check how you slice audio and keep sending. Audio only.invalid_image_frame: the frame was empty, was not a JPEG, or exceeded the pixel limit. Check the image and keep sending. Video only.message_invalid: a text message was not valid JSON, had notype, or had an unrecognizedtype. Correct the message and keep sending.rate_limited: media arrived faster than the send rate and was not processed. Audio: wait briefly and send the same frame again, so that the stream stays continuous. Video: from a live camera, drop the frame and send the next; from a recording, wait briefly and send the same frame again.
Depends on the endpoint.
internal_error: the server failed while processing media. Audio: the session ends as for a session-ending error; open a new connection. Video: the frame is skipped and the session continues; resend it if you need a result for it. Five consecutive failures end the session; frames that are rejected or have no faces to measure do not reset the count.
Session-ending. error is followed by session.closed with reason error, and the connection closes with code 1008.
config_invalid: asession.updatecontained an unrecognized field or an unsupported value, or arrived after you started sending media. Correct the message and open a new connection.message_too_large: a message exceeded the size limit. Reduce the frame size and open a new connection.
A human-readable description of the error, intended for logging and debugging. The wording may change; use code for programmatic handling.
false for every error that ends the session. true for every error the session survives, except internal_error on the video endpoint, where it reports whether the failure was temporary and the frame is worth resending.
Sent as the last message of the session. Carries the reason the session ended and totals of what was received and produced. The connection closes immediately after it, with the close code for the reason.
Unique identifier for the session. It is also the run_id of the run the session is recorded as. Include it when contacting support about a session.
Why the session ended. session.closed is the last message, and the WebSocket close code that follows it depends on the reason.
client_request: you sentsession.close. Close code 1000.server_shutdown: the server is restarting; reconnect at once to start a new session. Close code 1012.idle_timeout: the server received no media frame it could use for 30 seconds; open a new session to continue. Close code 1000.error: a session-ending error, reported in the precedingerrormessage. Close code 1008.