Skip to navigation

Realtime decisions

Ask questions about a live call as it happens, and use the answers to shape your voice agent's next reply.

This guide shows you how to get answers about live audio, such as a phone call or a microphone stream, so your voice agent or app can respond to what a transcript leaves out: a hesitant tone, a second voice, or a voicemail beep. Your code or framework cuts the audio into short clips and sends each one to Hume Audio Decisions with your questions. Each answer can shape when and how the agent replies.

How a realtime request works

Most systems that handle live speech run voice activity detection (VAD), which reports where speech starts and stops. Voice agents add turn detection, which decides whether a pause ends the caller’s turn.

Who cuts the clips

Hume Audio Decisions answers one clip per request and does not split a stream, so something in your stack cuts the clips. Depending on your framework, the clips may already be cut for you. The SDKs and cookbooks call a clip a chunk.

If you useThe clips are cut byYou choose
Pipecat’s PerceptionReaderThe reader, each time the caller pausesThe questions, and how long the reply waits for the answers. See Pipecat.
The Answering live audio cookbook appThe app, every 5 seconds from the microphoneThe questions, and the clip length, which CHUNK_MS sets.
LiveKitA small custom plugin, each time the caller pausesThe questions, and how long the reply waits for the answers. See LiveKit. Official LiveKit support is coming soon.
Twilio or your own pipelineYour codeWhich audio a clip holds, when to send it, and whether anything waits for the answer. See the checklist for other frameworks.

Audio length and response time

Every request has an audio length and a response time:

  • Audio length is how much audio a request carries, from 1 second to 5 minutes. You choose it when you cut the clip, and it decides what the answer is about.
  • Response time is the time from sending a request to getting its answer back: 0.22 seconds in Hume’s benchmark, plus network time.

A 5-second clip covers more of what the caller said than a 1-second clip. The two clips have the same response time. Only the response time can delay anything waiting on the answer; a clip that is already recorded when you send it adds no delay, whatever its length.

A clip that is still being recorded goes out when it ends, so a fixed 5-second window is sent 5 seconds after it starts. Anything waiting on the answer waits only for the response time, and in a voice agent only for the part that runs past speech-to-text and turn detection, as Choose whether the reply waits shows.

Choose which audio to send

Hume Audio Decisions answers each request from its clip alone, so the clip has to hold what the question is about.

Speech segmentsRaw stream
When your code cuts a clipWhen VAD, or a speech-to-speech API’s speech events, report the end of speechOn a timer, or from a fixed start, as in Choose when to send
What a clip holdsAudio around detected speech, including short pauses and any background sound at the same time. Longer gaps between segments are left out.Everything on the line: speech, silence, ringing, beeps, hold music, background noise
Use it forQuestions about how someone speaksQuestions about sounds other than speech, or about the line before anyone speaks

For speech segments:

  1. Choose whose voice. For questions about how one speaker sounds, send only that speaker’s audio, taken from their own input track before it is mixed with other voices, such as a voice agent’s. For questions about the whole conversation, send all sides mixed into one clip.
  2. Send at least 5 seconds where you can. Extend a shorter clip with the speaker’s earlier speech from the same turn, never with silence. If the clip is still shorter than 1 second, skip the request and continue without an answer, since Hume Audio Decisions rejects audio under 1 second with audio_too_short. See the live audio guidance.
  3. Do not join speech across another speaker’s turns. A “Yes.” in answer to a question carries its own tone. Joined to the speaker’s previous sentence, the “Yes.” tends to get the same answer as that sentence.

Keep or skip silence

Pauses carry meaning: how long someone waits before answering, and where they stop mid-sentence. A speech segment keeps the pauses inside it but drops the gaps between segments, so pick the audio by what the question asks.

The question is aboutPauses in the clipA clip that is mostly silence
How the speaker sounds, such as tone, pace, or hesitationSpeech segments work. Pauses inside a segment keep their real length.Skip it. There is no voice to judge.
Pauses themselves, such as “Does the speaker pause for half a second or more within their turn?”Send the continuous raw audio of the whole turn, cut on a timer or shortly after the speaker stops.Send it.
Sounds other than speech, such as ringing or a beepSend the raw stream as it arrived.Send it.

Choose when to send

When you send decides how much of the call each answer covers and how soon you get it. Pick the strategy that fits your question. Each clip can hold up to 5 minutes of audio.

StrategyA clip goes outEach answer coversUse it for
End of speechEach time VAD reports that the speaker stoppedThe latest speech, extended to 5 seconds within the turnHow the speaker said something, such as “Does the caller sound frustrated?”
Fixed windowsOn a timer, such as every 5 secondsThe last window, from the raw stream or from speech segmentsSounds on the line and checks that run for the whole call, such as “Is background noise making the speaker hard to hear?”
Expanding windowAt 1, 2, then 4 seconds from a fixed start, until the answer is clearEverything since the startOne early decision, such as “Is this a voicemail greeting?”
SlicesWhen a stretch of audio ends, or on a timerOne short slice, compared with its neighborsFinding where an event happens, such as “Does someone laugh?”

Send at the end of speech

Sending at the end of speech gets an answer about each thing the speaker says, often before the agent replies. Send a clip each time VAD reports that the speaker stopped. VAD ends a segment after its own stretch of silence, so the clip ends at a pause, though the speaker may continue the same sentence after the pause.

  1. Trigger: the VAD’s end of speech, or the speech-stopped event of a speech-to-speech API.
  2. Audio: the latest speech, extended to 5 seconds within the turn. For speech longer than 5 minutes, send the latest 5 minutes.
  3. Requests: one at a time per call. If the speaker pauses again while a request runs, hold only the newest clip and send it when the request finishes. Count the wait limit from the newest end of speech.

Send fixed windows

Your code cuts a window on a timer, whatever the speaker is doing, so fixed windows suit questions about the line itself, such as background noise or a second voice, and checks that run for the whole call.

Window length and how often you send a window are separate choices. A 5-second window sent every second gives a new answer each second, each about the latest 5 seconds. Overlapping windows keep a fresh answer ready for a reply that doesn’t wait for it, but they send five times as many requests.

  1. Trigger: a timer, such as every 5 seconds, or every second for overlapping windows.
  2. Audio: the last window, from the raw stream or from speech segments. A 5-second window of speech segments can hold less than 5 seconds, because gaps are left out, so extend it to 5 seconds before you send it.
  3. Requests: one per window per call. If the previous request is still running when the next window is due, skip that window. After a failed request, continue with the next window, and follow Retry-After on a 429. See Plan requests and billing for how many calls the rate limit covers.
  4. Example: ask “Is background noise making the speaker hard to hear?” and show a connection warning when the answer passes your threshold on two windows in a row.

Send an expanding window

An expanding window gets one decision as early as the audio allows, such as whether an outbound call reached a person or a voicemail greeting. Send 1 second from a fixed start, such as when the call connects, then 2 seconds from the same start, then 4.

  1. Start length: the shortest clip that gives reliable answers on your own test calls. Voicemail detection needs an early decision, so this example starts at 1 second and doubles. Most questions need 5 seconds; each use case lists its minimum.
  2. Audio: everything from the fixed start, doubling in length each time, up to a maximum of 5 minutes or less.
  3. Requests: one at a time. To handle each answer:
    1. Set an upper and a lower threshold on your own calls.
    2. Above the upper threshold, stop sending and wait for the voicemail greeting to end, then leave your message.
    3. Below the lower threshold, start the conversation.
    4. Between them, send the next, doubled window.
    5. At the maximum length, or when a request fails, use your fallback, such as treating the call as a person.

Send overlapping slices

Slices find where in the audio an event happens, such as a laugh. They use the same overlapping windows as fixed windows, and you compare their answers:

  1. Cut the audio into short overlapping slices.
  2. Ask every slice the same question.
  3. Compare each answer with a threshold you set on your own calls. A passing slice marks a likely event somewhere inside that slice.
  4. Merge neighboring passing slices into one detected stretch.
  5. Check your detections against labeled recordings.
  1. Trigger: the end of the stretch you want to examine, or a timer.
  2. Audio: make each slice at least twice as long as the longest event you expect, and start a new slice at least every half slice. With that spacing, every event of that length fits whole in one slice. Use slices shorter than 5 seconds only for questions that give reliable answers at that length, such as whether someone laughs.
  3. Requests: send the slices one after another, oldest first. Each counts toward the rate limit.

Voice agents

In a voice agent, answers from Hume Audio Decisions can change the agent’s next reply or trigger an action in your backend, without replacing any part of your pipeline.

Where Hume Audio Decisions fits

In a cascade

Speech-to-text, an LLM, and text-to-speech run one after another, and Hume Audio Decisions runs beside them. A request sent when VAD reports the end of speech runs while turn detection decides whether the turn is over, so an answer the reply waits for often arrives before the LLM would start.

A cascade with Hume Audio Decisions

Caller audioSpeech-to-textVADTurn detectionLLMText-to-speechAgent audioHume Audio Decisionsraw streamspeech segmentsinline: the reply waitsYour backendactions, alerts, later turnsbackground: no wait
The solid arrow from Hume Audio Decisions into the LLM is the inline path: the answer reaches the LLM before it writes the reply. The dashed arrow is the background path: the answer goes to your backend without holding the reply. An agent can use both paths by sending one request with all its questions, then using some answers inline and passing the rest to your backend.

With a speech-to-speech model

One model takes the caller’s audio and returns the reply as audio, so there is no transcript step for the request to run beside.

A speech-to-speech model with Hume Audio Decisions

Caller audioSpeech-to-speech modelAgent audioHume Audio Decisionsraw stream or speech segmentsinline: the reply waitsYour backendactions, alerts, later turnsbackground: no wait
The solid arrow from Hume Audio Decisions into the model is the inline path: the model’s response waits until the answer is added. The dashed arrow is the background path: the answer goes to your backend or into a later response.
  • The model’s own speech events can cut speech segments without a VAD of your own. The OpenAI Realtime API, for example, sends input_audio_buffer.speech_started and input_audio_buffer.speech_stopped.
  • To use an answer on the current response, hold the response until the answer arrives, as in Inline.
  • A background answer goes into the conversation for a later response, or to your backend.

Choose whether the reply waits

A reply that waits can respond to how the caller sounded on this turn. A reply that doesn’t wait adds no delay and uses the answer later. Turn detection, speech-to-text, the LLM, and text-to-speech all run between the caller stopping and the agent speaking. Inline and background requests both run alongside these steps. An inline request also holds the LLM until the answer arrives.

Inline: the reply waits for the answer

Caller
Stops 3 s
Request
3.52 s
Turn detection
3.8 s
Reply
Uses it
0 s35 s

Step 4 of 4: Turn detection ends the turn at 3.8 seconds. The answer is already in, so the reply starts on time and responds to how the caller sounded.

Example times. VAD waits 0.3 s of silence before it reports the end of speech, and turn detection takes 0.5 s here; yours may differ. The response time, between the dashed lines, is 0.22 s. If an answer arrives after turn detection, the reply waits for it, up to the wait limit.

Background: the reply starts without waiting

Caller
Stops 3 s
Request
3.52 s
Turn detection
3.8 s
Your backend
Alert
Reply
No wait
0 s35 s

Step 4 of 4: Turn detection ends the turn at 3.8 seconds, and the reply starts without waiting. An answer that arrives later reaches the next turn.

Example times. VAD waits 0.3 s of silence before it reports the end of speech, and turn detection takes 0.5 s here; yours may differ. The response time, between the dashed lines, is 0.22 s.

InlineBackground
The LLM waits for the answerYesNo
The answer is usedOn the reply to the same speechOn the current reply if it has already arrived when the LLM starts, otherwise on a later turn or in your backend
Use it forAdapting the reply to how the caller sounded, deciding whether to start the conversationTool calls and other actions in your backend, alerts, moderation, context for later turns, call notes

Inline

The LLM writes its reply once it has both the transcript and the answer about the same speech, so the reply can respond to how the caller sounded, such as offering to slow down when “I’m fine” sounds sad. Pipecat’s PerceptionReader works this way.

  1. Send the clip as soon as VAD reports the end of speech. See Choose when to send.

  2. Hold the LLM until the answer arrives, up to a wait limit. Pipecat’s default is 2 seconds after the caller stops. The response time is 0.22 seconds, so the limit takes effect only when a request fails or the network is slow. If the limit passes, reply without the answer.

  3. In a cascade, put the answer at the head of the caller’s message to the LLM, such as [tone sad 0.81] I'm fine. Let's just get the new booking done., and say what it means in the system prompt, such as tone: the likeliest tones, each with the estimated share of raters who would choose it.

  4. With a speech-to-speech model, hold the model’s response yourself. In the OpenAI Realtime API, for example:

    1. Set create_response to false in turn_detection, so the model does not respond on its own at the end of the turn.
    2. When the answer arrives, send conversation.item.create with a message holding the answer line.
    3. Send response.create to start the response.
    4. At the wait limit, send response.create without the answer.

    Without the hold, the model starts replying at the speech-stopped event, so holding the reply adds the full response time.

Background

A background request adds no time to the reply.

  1. Send the clip as soon as VAD reports the end of speech, and start the LLM without waiting.
  2. Tag each request with the turn it came from. When the answer arrives, add it to the context with that turn, such as On turn 12, the caller sounded frustrated (0.82)., and drop answers from turns older than the last one or two.
  3. To act in your backend instead, call a tool, update a record, hand the call to a person, raise an alert past a threshold, or save the answer with the call.

Background answers on the current reply

A background answer reaches the current reply when it arrives before the LLM starts. Send overlapping windows while the caller speaks, such as the latest 5 seconds every second.

  1. When the turn ends, add the newest answer that has already arrived to the caller’s message. The reply does not wait.
  2. That answer’s window ends before the turn does, so it can leave out the caller’s last words.
  3. Overlapping windows send five times the requests of back-to-back windows. See Plan requests and billing.

Set the wait limit in code

An inline reply waits for the request, so give it a wait limit:

  1. Count the limit from when the caller stopped. VAD reports the end of speech after its own stretch of silence, such as Silero’s stop_secs, so subtract that silence from the event’s timestamp, as Pipecat does.
  2. Send the request once. By default, the Python and TypeScript SDKs wait up to 60 seconds and retry a failed request twice.
  3. The Python and TypeScript examples stop waiting 2 seconds after the caller stops. The cURL example allows 2 seconds from when the request goes out; to hold it to the same limit, work out the time left in your application and pass that to --max-time.
curl https://api.cloud.hume.ai/v0/decisions/file \
--max-time 2 \
-H "X-Hume-Api-Key: $HUME_API_KEY" \
-F 'request={
"questions": {
"hesitant": {
"type": "noul",
"instructions": "Does the caller sound hesitant or unsure?"
}
}
};type=application/json' \
-F file=@clip.wav
  1. A request stopped at the wait limit returns no answer. To use a late answer on a later turn instead, release the reply at the limit but let the request run on in its own task, and add its answer when it arrives.
  2. For requests sent on a timer or in an expanding window, count the wait limit from each send, and also set a maximum listening time, after which you use your fallback, as in Send an expanding window.

Common setups

GoalReplyAudioWhen to send
Adapt the reply to how the caller soundedInlineSpeech segmentsEnd of speech
Adapt a speech-to-speech model’s replyInlineSpeech segments, cut at the model’s speech eventsEnd of speech
Tell a voicemail greeting from a person on an outbound callInline: the agent’s first words wait for the answerRaw streamExpanding window
Alert a supervisor or call a tool when a caller sounds upsetBackgroundSpeech segmentsEnd of speech
Notice background noise or a second voice on the lineBackgroundRaw streamFixed windows
Find when the caller laughsBackgroundRaw streamSlices

Checklist for other frameworks

Pipecat handles these steps for you. In any other framework, or an agent you build yourself, your code needs to:

StepHow
Copy the caller’s audioCopy each frame where the agent receives it and pass the frame on unchanged, so speech-to-text gets it without delay.
Cut a clipKeep a rolling buffer of the incoming audio. At the trigger you chose in Choose when to send, cut the clip and convert it to 16 kHz, 16-bit mono WAV. Hume Audio Decisions processes audio at 16 kHz, so a higher rate only makes the upload larger; a 5-second clip is about 160 KB. For speech segments, take the latest speech with up to 1 second before it and extend it to 5 seconds with earlier speech from the same turn. For the raw stream, keep the whole interval, including silence and other sounds.
Clear the bufferFor speech segments, drop earlier speech when the agent’s turn ends. On a mute, drop it and stop collecting, so no clip joins speech from before and after the mute.
Send the requestFrom a background task, so audio keeps flowing while the request runs, with one client reused for every clip so requests share an open connection, one request at a time per call, with a wait limit on inline requests.
Use the answers inlineHold the LLM until the answer arrives or the wait limit passes, then add the answer to the caller’s message, or reply without it.
Use the answers in the backgroundStart the LLM as usual, with any answer that has already arrived. Add later answers to the context for the next turn, or send them to your backend.
Keep late answersRelease the reply at the wait limit but let the request run on in its own task, so its answer can still reach a later turn.

Other live audio

Outside a voice agent, answers feed a dashboard, an alert, or on-screen feedback, so nothing waits for them: send each clip as it is cut and act on each answer as it arrives.

AudioWhat to askHow to cut the clips
A room, a device, or a broadcastWhether a named sound is heard, such as an alarm or running waterSlices of the raw stream, so each sound fits whole in at least one
Calls between peopleHow each person sounds, with the person named in the question, so a supervisor can see which calls need attentionFixed windows, with both sides mixed into one clip
Speaking practiceSpeaking rate or filler words, shown to the speaker while they talkFixed 5-second windows from the microphone, as in the Answering live audio cookbook
Call and stream qualityWhether the audio cuts out or there is noise behind the speakerFixed windows of the audio as the listener receives it

When every clip needs an answer, as in the Answering live audio app, requests overlap, so count them against the rate limit, and keep each answer with its own clip, since answers can arrive out of order.

Choose questions

  1. Ask about what a listener hears in the clip. “Does the caller sound hesitant or unsure?” and “Are several people talking?” are answered from the audio. “Who is this caller?” is not.
  2. Keep inline questions few. Each answer adds a line to every caller message the LLM reads. Start with one noul question, such as “Does the caller sound frustrated?”, and add more once you have seen its answers on your own calls.
  3. Send the same questions, in the same order, with every clip. All the questions in a request are answered together, so an answer can change when a question is added, removed, or moved. Keeping the questions fixed keeps answers comparable across turns. See Writing questions.
  4. Describe the call in context. For example, An outbound call from an airline's rebooking agent. The caller is the customer. A transcript of the turn can go on its own line, labeled Transcript:. With Pipecat, PerceptionReader always sets context to the bot’s last reply, so you cannot add your own description; see Pipecat.

Act on answers

Hume Audio Decisions answers each clip on its own and doesn’t combine answers across clips, so your code does.

  1. Tell the agent how to use each answer. Report a noul answer as its value and a choice answer as its likeliest options, and say what to do past a threshold, such as If hesitation is 0.7 or above, check that the caller is following.
  2. Set thresholds on clips cut the way you will cut them live. The same question gives a different value on one turn than on a whole call, so record and label clips from your own calls, then set each threshold as in Setting thresholds.
  3. For a decision about the whole call, require the threshold to pass on two or three clips in a row. One clip covers a few seconds, so wait for a run of passing answers before handing a call to a person, for example.
  4. Store each answer as it arrives, with its run_id and the start and end of its clip in the call. Hume Audio Decisions returns each answer only once, in its response. See Storing answers.

Plan requests and billing

Every API key in an organization shares a limit of 600 requests per minute, across every call at once. Each request is billed by its input tokens, which come mostly from its audio, so audio you send in two clips is billed twice. Estimate input tokens takes an audio duration from 1 second to 5 minutes and your questions, and returns the count before you send any audio.

ScheduleRequests per call per minuteCalls that reach 600 per minuteHow often the same audio is sent
End of speech, at 6 pauses a minute6100Once, plus earlier speech re-sent to extend a short clip
5-second windows, one after another1250Once
5-second windows, one sent every second6010Up to five times

Next steps