Realtime decisions
This guide shows you how to get answers about live audio, such as a phone call or a microphone stream, so your voice agent or app can respond to what a transcript leaves out: a hesitant tone, a second voice, or a voicemail beep. Your code or framework cuts the audio into short clips and sends each one to Hume Audio Decisions with your questions. Each answer can shape when and how the agent replies.
How a realtime request works
Most systems that handle live speech run voice activity detection (VAD), which reports where speech starts and stops. Voice agents add turn detection, which decides whether a pause ends the caller’s turn.
Who cuts the clips
Hume Audio Decisions answers one clip per request and does not split a stream, so something in your stack cuts the clips. Depending on your framework, the clips may already be cut for you. The SDKs and cookbooks call a clip a chunk.
Audio length and response time
Every request has an audio length and a response time:
- Audio length is how much audio a request carries, from 1 second to 5 minutes. You choose it when you cut the clip, and it decides what the answer is about.
- Response time is the time from sending a request to getting its answer back: 0.22 seconds in Hume’s benchmark, plus network time.
A 5-second clip covers more of what the caller said than a 1-second clip. The two clips have the same response time. Only the response time can delay anything waiting on the answer; a clip that is already recorded when you send it adds no delay, whatever its length.
A clip that is still being recorded goes out when it ends, so a fixed 5-second window is sent 5 seconds after it starts. Anything waiting on the answer waits only for the response time, and in a voice agent only for the part that runs past speech-to-text and turn detection, as Choose whether the reply waits shows.
Two recorded clips sent together
Choose which audio to send
Hume Audio Decisions answers each request from its clip alone, so the clip has to hold what the question is about.
For speech segments:
- Choose whose voice. For questions about how one speaker sounds, send only that speaker’s audio, taken from their own input track before it is mixed with other voices, such as a voice agent’s. For questions about the whole conversation, send all sides mixed into one clip.
- Send at least 5 seconds where you can. Extend a shorter clip with the speaker’s earlier speech from the same turn, never with silence. If the clip is still shorter than 1 second, skip the request and continue without an answer, since Hume Audio Decisions rejects audio under 1 second with
audio_too_short. See the live audio guidance. - Do not join speech across another speaker’s turns. A “Yes.” in answer to a question carries its own tone. Joined to the speaker’s previous sentence, the “Yes.” tends to get the same answer as that sentence.
Keep or skip silence
Pauses carry meaning: how long someone waits before answering, and where they stop mid-sentence. A speech segment keeps the pauses inside it but drops the gaps between segments, so pick the audio by what the question asks.
Choose when to send
When you send decides how much of the call each answer covers and how soon you get it. Pick the strategy that fits your question. Each clip can hold up to 5 minutes of audio.
Send at the end of speech
Sending at the end of speech gets an answer about each thing the speaker says, often before the agent replies. Send a clip each time VAD reports that the speaker stopped. VAD ends a segment after its own stretch of silence, so the clip ends at a pause, though the speaker may continue the same sentence after the pause.
Clips cut at the speaker’s pauses
- Trigger: the VAD’s end of speech, or the speech-stopped event of a speech-to-speech API.
- Audio: the latest speech, extended to 5 seconds within the turn. For speech longer than 5 minutes, send the latest 5 minutes.
- Requests: one at a time per call. If the speaker pauses again while a request runs, hold only the newest clip and send it when the request finishes. Count the wait limit from the newest end of speech.
Send fixed windows
Your code cuts a window on a timer, whatever the speaker is doing, so fixed windows suit questions about the line itself, such as background noise or a second voice, and checks that run for the whole call.
5-second windows on the raw stream
Window length and how often you send a window are separate choices. A 5-second window sent every second gives a new answer each second, each about the latest 5 seconds. Overlapping windows keep a fresh answer ready for a reply that doesn’t wait for it, but they send five times as many requests.
5-second windows, one sent every second
- Trigger: a timer, such as every 5 seconds, or every second for overlapping windows.
- Audio: the last window, from the raw stream or from speech segments. A 5-second window of speech segments can hold less than 5 seconds, because gaps are left out, so extend it to 5 seconds before you send it.
- Requests: one per window per call. If the previous request is still running when the next window is due, skip that window. After a failed request, continue with the next window, and follow
Retry-Afteron a429. See Plan requests and billing for how many calls the rate limit covers. - Example: ask “Is background noise making the speaker hard to hear?” and show a connection warning when the answer passes your threshold on two windows in a row.
Send an expanding window
An expanding window gets one decision as early as the audio allows, such as whether an outbound call reached a person or a voicemail greeting. Send 1 second from a fixed start, such as when the call connects, then 2 seconds from the same start, then 4.
Windows that grow from the start of the call, asked “Is this a voicemail greeting?”
- Start length: the shortest clip that gives reliable answers on your own test calls. Voicemail detection needs an early decision, so this example starts at 1 second and doubles. Most questions need 5 seconds; each use case lists its minimum.
- Audio: everything from the fixed start, doubling in length each time, up to a maximum of 5 minutes or less.
- Requests: one at a time. To handle each answer:
- Set an upper and a lower threshold on your own calls.
- Above the upper threshold, stop sending and wait for the voicemail greeting to end, then leave your message.
- Below the lower threshold, start the conversation.
- Between them, send the next, doubled window.
- At the maximum length, or when a request fails, use your fallback, such as treating the call as a person.
Send overlapping slices
Slices find where in the audio an event happens, such as a laugh. They use the same overlapping windows as fixed windows, and you compare their answers:
- Cut the audio into short overlapping slices.
- Ask every slice the same question.
- Compare each answer with a threshold you set on your own calls. A passing slice marks a likely event somewhere inside that slice.
- Merge neighboring passing slices into one detected stretch.
- Check your detections against labeled recordings.
2-second slices that start every second, asked “Does someone laugh?”
- Trigger: the end of the stretch you want to examine, or a timer.
- Audio: make each slice at least twice as long as the longest event you expect, and start a new slice at least every half slice. With that spacing, every event of that length fits whole in one slice. Use slices shorter than 5 seconds only for questions that give reliable answers at that length, such as whether someone laughs.
- Requests: send the slices one after another, oldest first. Each counts toward the rate limit.
Voice agents
In a voice agent, answers from Hume Audio Decisions can change the agent’s next reply or trigger an action in your backend, without replacing any part of your pipeline.
Where Hume Audio Decisions fits
In a cascade
Speech-to-text, an LLM, and text-to-speech run one after another, and Hume Audio Decisions runs beside them. A request sent when VAD reports the end of speech runs while turn detection decides whether the turn is over, so an answer the reply waits for often arrives before the LLM would start.
A cascade with Hume Audio Decisions
With a speech-to-speech model
One model takes the caller’s audio and returns the reply as audio, so there is no transcript step for the request to run beside.
A speech-to-speech model with Hume Audio Decisions
- The model’s own speech events can cut speech segments without a VAD of your own. The OpenAI Realtime API, for example, sends
input_audio_buffer.speech_startedandinput_audio_buffer.speech_stopped. - To use an answer on the current response, hold the response until the answer arrives, as in Inline.
- A background answer goes into the conversation for a later response, or to your backend.
Choose whether the reply waits
A reply that waits can respond to how the caller sounded on this turn. A reply that doesn’t wait adds no delay and uses the answer later. Turn detection, speech-to-text, the LLM, and text-to-speech all run between the caller stopping and the agent speaking. Inline and background requests both run alongside these steps. An inline request also holds the LLM until the answer arrives.
Inline: the reply waits for the answer
Step 4 of 4: Turn detection ends the turn at 3.8 seconds. The answer is already in, so the reply starts on time and responds to how the caller sounded.
Example times. VAD waits 0.3 s of silence before it reports the end of speech, and turn detection takes 0.5 s here; yours may differ. The response time, between the dashed lines, is 0.22 s. If an answer arrives after turn detection, the reply waits for it, up to the wait limit.
Background: the reply starts without waiting
Step 4 of 4: Turn detection ends the turn at 3.8 seconds, and the reply starts without waiting. An answer that arrives later reaches the next turn.
Example times. VAD waits 0.3 s of silence before it reports the end of speech, and turn detection takes 0.5 s here; yours may differ. The response time, between the dashed lines, is 0.22 s.
Inline
The LLM writes its reply once it has both the transcript and the answer about the same speech, so the reply can respond to how the caller sounded, such as offering to slow down when “I’m fine” sounds sad. Pipecat’s PerceptionReader works this way.
-
Send the clip as soon as VAD reports the end of speech. See Choose when to send.
-
Hold the LLM until the answer arrives, up to a wait limit. Pipecat’s default is 2 seconds after the caller stops. The response time is 0.22 seconds, so the limit takes effect only when a request fails or the network is slow. If the limit passes, reply without the answer.
-
In a cascade, put the answer at the head of the caller’s message to the LLM, such as
[tone sad 0.81] I'm fine. Let's just get the new booking done., and say what it means in the system prompt, such astone: the likeliest tones, each with the estimated share of raters who would choose it. -
With a speech-to-speech model, hold the model’s response yourself. In the OpenAI Realtime API, for example:
- Set
create_responsetofalseinturn_detection, so the model does not respond on its own at the end of the turn. - When the answer arrives, send
conversation.item.createwith a message holding the answer line. - Send
response.createto start the response. - At the wait limit, send
response.createwithout the answer.
Without the hold, the model starts replying at the speech-stopped event, so holding the reply adds the full response time.
- Set
Background
A background request adds no time to the reply.
- Send the clip as soon as VAD reports the end of speech, and start the LLM without waiting.
- Tag each request with the turn it came from. When the answer arrives, add it to the context with that turn, such as
On turn 12, the caller sounded frustrated (0.82)., and drop answers from turns older than the last one or two. - To act in your backend instead, call a tool, update a record, hand the call to a person, raise an alert past a threshold, or save the answer with the call.
Background answers on the current reply
A background answer reaches the current reply when it arrives before the LLM starts. Send overlapping windows while the caller speaks, such as the latest 5 seconds every second.
- When the turn ends, add the newest answer that has already arrived to the caller’s message. The reply does not wait.
- That answer’s window ends before the turn does, so it can leave out the caller’s last words.
- Overlapping windows send five times the requests of back-to-back windows. See Plan requests and billing.
Set the wait limit in code
An inline reply waits for the request, so give it a wait limit:
- Count the limit from when the caller stopped. VAD reports the end of speech after its own stretch of silence, such as Silero’s
stop_secs, so subtract that silence from the event’s timestamp, as Pipecat does. - Send the request once. By default, the Python and TypeScript SDKs wait up to 60 seconds and retry a failed request twice.
- The Python and TypeScript examples stop waiting 2 seconds after the caller stops. The cURL example allows 2 seconds from when the request goes out; to hold it to the same limit, work out the time left in your application and pass that to
--max-time.
- A request stopped at the wait limit returns no answer. To use a late answer on a later turn instead, release the reply at the limit but let the request run on in its own task, and add its answer when it arrives.
- For requests sent on a timer or in an expanding window, count the wait limit from each send, and also set a maximum listening time, after which you use your fallback, as in Send an expanding window.
Common setups
Checklist for other frameworks
Pipecat handles these steps for you. In any other framework, or an agent you build yourself, your code needs to:
Other live audio
Outside a voice agent, answers feed a dashboard, an alert, or on-screen feedback, so nothing waits for them: send each clip as it is cut and act on each answer as it arrives.
When every clip needs an answer, as in the Answering live audio app, requests overlap, so count them against the rate limit, and keep each answer with its own clip, since answers can arrive out of order.
Choose questions
- Ask about what a listener hears in the clip. “Does the caller sound hesitant or unsure?” and “Are several people talking?” are answered from the audio. “Who is this caller?” is not.
- Keep inline questions few. Each answer adds a line to every caller message the LLM reads. Start with one
noulquestion, such as “Does the caller sound frustrated?”, and add more once you have seen its answers on your own calls. - Send the same questions, in the same order, with every clip. All the questions in a request are answered together, so an answer can change when a question is added, removed, or moved. Keeping the questions fixed keeps answers comparable across turns. See Writing questions.
- Describe the call in
context. For example,An outbound call from an airline's rebooking agent. The caller is the customer.A transcript of the turn can go on its own line, labeledTranscript:. With Pipecat,PerceptionReaderalways setscontextto the bot’s last reply, so you cannot add your own description; see Pipecat.
Act on answers
Hume Audio Decisions answers each clip on its own and doesn’t combine answers across clips, so your code does.
- Tell the agent how to use each answer. Report a
noulanswer as its value and achoiceanswer as its likeliest options, and say what to do past a threshold, such asIf hesitation is 0.7 or above, check that the caller is following. - Set thresholds on clips cut the way you will cut them live. The same question gives a different value on one turn than on a whole call, so record and label clips from your own calls, then set each threshold as in Setting thresholds.
- For a decision about the whole call, require the threshold to pass on two or three clips in a row. One clip covers a few seconds, so wait for a run of passing answers before handing a call to a person, for example.
- Store each answer as it arrives, with its
run_idand the start and end of its clip in the call. Hume Audio Decisions returns each answer only once, in its response. See Storing answers.
Plan requests and billing
Every API key in an organization shares a limit of 600 requests per minute, across every call at once. Each request is billed by its input tokens, which come mostly from its audio, so audio you send in two clips is billed twice. Estimate input tokens takes an audio duration from 1 second to 5 minutes and your questions, and returns the count before you send any audio.