For coding agents

This page gives a coding agent what it needs to build with the Expression Measurement API: what the API does, one complete request and response, the realtime session flow and the rules that code most often gets wrong. Point your agent to this page before it writes code that calls the API.

Set up your agent

Add this to your project’s agent instructions, such as CLAUDE.md or AGENTS.md:

Before writing code that calls the Expression Measurement API, read https://dev.hume.ai/expression-measurement/docs/for-coding-agents.md. For any other Expression Measurement page, find it in https://dev.hume.ai/expression-measurement/llms.txt and fetch its Markdown by adding .md to the page URL.

What the API does

1. The API measures emotional expression in speech and in faces. The audio endpoints detect utterances in speech and measure each window of speech. The video endpoints detect faces in JPEG images and measure the largest faces in each one.
2. Each measurement is lists of scores, not text. A score pairs a name, such as amusement, with a probability from 0 to 1. Each array includes only the scores that pass its own cutoff.
3. Upload endpoints measure media you already have, and realtime endpoints measure media as it streams. An upload is one multipart/form-data request that returns every measurement in its response. A realtime session is a WebSocket that returns each measurement as soon as it is produced. For the same audio, both produce the same measurements.
4. Every upload request and realtime session is recorded as a run. The run endpoints return a run’s status, configuration and event log, but not its measurements.

Make a request

An audio upload is a multipart/form-data POST to https://api.cloud.hume.ai/v1/expression/audio/file, with the API key in the X-Hume-Api-Key header and up to two parts:

1. file: one audio file, as a 16 kHz mono 16-bit PCM WAV file or as headerless samples in that format.
2. config: an optional JSON object of settings, such as measurement_timer_ms, the interval between measurements.

curl https://api.cloud.hume.ai/v1/expression/audio/file \
  -H "X-Hume-Api-Key: $HUME_API_KEY" \
  -F "file=@speech.wav;type=audio/wav" \
  -F 'config={
    "measurement_timer_ms": 5000
  };type=application/json'

The response lists every utterance with its span, and every measurement with the utterance_id it belongs to. Times are in milliseconds from the start of the audio. This response is for 5 seconds of audio containing one utterance:

{
  "request_id": "3f9a1c07e2b84d56a0c7e19b5d2f8a64",
  "run_id": "01926f3c-d33c-7c9f-9c91-4502d80834d5",
  "received": {
    "audio_duration_ms": 5000
  },
  "utterances": [
    {
      "utterance_id": 0,
      "audio_start_ms": 98,
      "audio_end_ms": 4510
    }
  ],
  "measurements": [
    {
      "measurement_id": 0,
      "utterance_id": 0,
      "audio_start_ms": 98,
      "audio_end_ms": 3098,
      "trigger": "interval",
      "expressions": [
        {
          "name": "interest",
          "probability": 0.4286
        },
        {
          "name": "curiosity",
          "probability": 0.308
        }
      ],
      "voice_attributes": [
        {
          "name": "clear",
          "probability": 0.9612
        },
        {
          "name": "calm",
          "probability": 0.9148
        }
      ]
    },
    {
      "measurement_id": 1,
      "utterance_id": 0,
      "audio_start_ms": 98,
      "audio_end_ms": 4510,
      "trigger": "speech_ended",
      "expressions": [
        {
          "name": "amusement",
          "probability": 0.5
        },
        {
          "name": "joy",
          "probability": 0.3593
        }
      ],
      "voice_attributes": [
        {
          "name": "cheerful",
          "probability": 0.9275
        },
        {
          "name": "bright",
          "probability": 0.8864
        }
      ]
    }
  ]
}

An image upload to https://api.cloud.hume.ai/v1/expression/video/file takes one or more JPEG images as file parts, and returns one result per image with the faces found in it. See Image upload.

Stream in realtime

The realtime endpoints are wss://api.cloud.hume.ai/v1/expression/audio/realtime and wss://api.cloud.hume.ai/v1/expression/video/realtime. Both use the same session protocol:

1. Open the WebSocket with the API key in the X-Hume-Api-Key header. The server sends session.created with the session_id, which is also the session’s run ID.
2. To change the defaults, send session.update before the first media frame. The server confirms with session.updated.
3. Send media as binary frames. On the audio endpoint, send headerless 16 kHz mono 16-bit PCM in frames of about 100 ms. On the video endpoint, send one JPEG image per frame.
4. Read each measurement.result. The audio endpoint also sends utterance.start and utterance.end around each stretch of speech. The video endpoint replies to every frame with a measurement.result or an error.
5. Send session.close, then read until session.closed arrives. Results still in progress are delivered before it, and it totals what the server received and produced.

See Sessions for the full protocol.

Score arrays

Each array’s probability denotes a different quantity, and each array has its own cutoff.

Rules for calling code

Scores

1. Treat each score as independent. Each probability is its own estimate for one name. Scores do not sum to 1, several names can score highly at once and they are not meant to be compared with each other.
2. Handle empty arrays. An empty array means that no score passed the cutoff, so a measurement may list fewer scores than expected, or none.
3. Do not hardcode the list of names. New names may be added, so treat a name you do not recognize as a new prediction. The audio and video endpoints use different expression names.
4. Read a face’s confidence as detection, not expression. It is the detector’s confidence that the bounding box contains a face.
5. Store the measurements you need. The run endpoints return a run’s status, configuration and event log, but not its measurements.

Audio

1. Send 16 kHz mono 16-bit PCM. Audio is not resampled or downmixed. A WAV file in another format is rejected with audio_decode_failed, and MP3, FLAC and other containers with unsupported_media_type. Convert with ffmpeg -i recording.mp3 -acodec pcm_s16le -ac 1 -ar 16000 speech.wav.
2. Keep an audio upload within 25 MB, a little over 13 minutes of audio. A larger request is rejected with audio_too_long.
3. Find where an utterance ends from utterances or utterance.end, not from trigger. The final measurement of an utterance can be skipped, so its last measurement can have a trigger of interval.
4. Stream audio no faster than 1 second of audio per second. A faster frame can be rejected with rate_limited; wait briefly and send the same frame again. To measure a recording faster than it plays, upload it instead.

Video

1. Send JPEG images of up to 2 MB and 3840x2160 pixels. Other formats are rejected.
2. Handle unmeasured faces. Only the largest faces are measured, 2 by default, and up to 32 are listed. A face that was not measured has face_id, expressions and descriptions set to null.
3. Match image upload results to images by position. measurements holds one result per image, in the order the file parts were sent.
4. Match realtime replies to frames by position, not by frame_id. Keep a queue of sent frames and remove the oldest for each measurement.result, and for each error whose code is invalid_image_frame, rate_limited or internal_error. frame_id advances only when a frame produces a result.
5. Stream at most 3 frames per second. A faster frame can be rejected with rate_limited. From a live camera, drop it and send the next frame. From a recording, wait briefly and send the same frame again.
6. Do not use face_id to identify a person. It links a face across the images of one request or session and restarts at 0 in each. To continue numbering across image uploads, pass face_state from one response to the next request.

Sessions

1. Include every setting you want in each session.update. It replaces the configuration rather than merging into it. The configuration locks at the first media frame, and a later session.update or an unrecognized field ends the session with config_invalid.
2. Close with session.close and read until session.closed. Closing the WebSocket instead loses the results in progress.
3. Keep media flowing. A session closes after 30 seconds without a media frame the server can use. To pause for longer, end the session and open a new one.
4. Open a new connection after any session ends. A session cannot be resumed. A close that was not preceded by session.closed means the session was lost. The Node.js SDK’s socket connects again after a server restart or a dropped connection, and so does the Python SDK’s reconnecting helper. Each reconnect starts a new session whose IDs and times start again from 0.

Requests and errors

1. Keep the API key on a server you control. Browsers cannot set the header on a WebSocket handshake, so relay media from a browser through your server.
2. Send only documented fields. A config part or session.update with an unrecognized field is rejected with config_invalid. In responses and messages, ignore fields you do not recognize.
3. Branch on the error’s code, and on the status when there is no code. A 401 is answered before the request reaches the service and carries no code. A realtime handshake rejected with 401 or 403 sends no error message, so read the status from the handshake result.
4. Fix a 4xx before resending, retry a 500 and wait for the seconds in Retry-After before retrying a 503. A realtime error carries retryable, which is false for every error that ends the session. See Errors.

Where to go next