> This page is for Audio Decisions.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://dev.hume.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://dev.hume.ai/_mcp/server.

# Realtime decisions

**This guide shows you how to get answers about live audio, such as a phone call or a microphone stream, so your voice agent or app can respond to what a transcript leaves out: a hesitant tone, a second voice, or a voicemail beep.** Your code or framework cuts the audio into short clips and sends each one to Hume Audio Decisions with your questions. Each answer can shape when and how the agent replies.

## How a realtime request works

Most systems that handle live speech run voice activity detection (VAD), which reports where speech starts and stops. Voice agents add turn detection, which decides whether a pause ends the caller's turn.

### Who cuts the clips

**Hume Audio Decisions answers one clip per request and does not split a stream, so something in your stack cuts the clips.** Depending on your framework, the clips may already be cut for you. The SDKs and cookbooks call a clip a chunk.

<table>
  <thead>
    <tr>
      <th width="24%">
        If you use
      </th>

      <th width="36%">
        The clips are cut by
      </th>

      <th>
        You choose
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        Pipecat's 

        `PerceptionReader`
      </td>

      <td>
        The reader, each time the caller pauses
      </td>

      <td>
        The questions, and how long the reply waits for the answers. See 

        [Pipecat](/audio-decisions/docs/integrations/pipecat)

        .
      </td>
    </tr>

    <tr>
      <td>
        The 

        [Answering live audio](/audio-decisions/cookbooks/answering-live-audio)

         cookbook app
      </td>

      <td>
        The app, every 5 seconds from the microphone
      </td>

      <td>
        The questions, and the clip length, which 

        `CHUNK_MS`

         sets.
      </td>
    </tr>

    <tr>
      <td>
        LiveKit
      </td>

      <td>
        A small custom plugin, each time the caller pauses
      </td>

      <td>
        The questions, and how long the reply waits for the answers. See 

        [LiveKit](/audio-decisions/docs/integrations/livekit)

        . Official LiveKit support is coming soon.
      </td>
    </tr>

    <tr>
      <td>
        Twilio or your own pipeline
      </td>

      <td>
        Your code
      </td>

      <td>
        Which audio a clip holds, when to send it, and whether anything waits for the answer. See the 

        [checklist for other frameworks](#checklist-for-other-frameworks)

        .
      </td>
    </tr>
  </tbody>
</table>

### Audio length and response time

Every request has an audio length and a response time:

* **Audio length** is how much audio a request carries, from 1 second to 5 minutes. You choose it when you cut the clip, and it decides what the answer is about.
* **Response time** is the time from sending a request to getting its answer back: 0.22 seconds in Hume's benchmark, plus network time.

A 5-second clip covers more of what the caller said than a 1-second clip. The two clips have the same response time. Only the response time can delay anything waiting on the answer; a clip that is already recorded when you send it adds no delay, whatever its length.

A clip that is still being recorded goes out when it ends, so a fixed 5-second window is sent 5 seconds after it starts. Anything waiting on the answer waits only for the response time, and in a voice agent only for the part that runs past speech-to-text and turn detection, as [Choose whether the reply waits](#choose-whether-the-reply-waits) shows.

Two recorded clips sent together

Speaker Stops 4.7 s

Audio length: 5 s

5-second clip

Audio sent

5.22 s

1 s

1-second clip 5.22 s

Response time: 0.22 s

0 s 5 8 s

Example times. The speaker stops at 4.7 s, and VAD reports the end of speech at 5 s, after its 0.3 s of silence. At 5 s, both clips are already recorded and are sent. The darker span between the dashed lines is the response time, the same 0.22 s for both.

## Choose which audio to send

Hume Audio Decisions answers each request from its clip alone, so the clip has to hold what the question is about.

<table>
  <thead>
    <tr>
      <th />

      <th>
        Speech segments
      </th>

      <th>
        Raw stream
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **When your code cuts a clip**
      </td>

      <td>
        When VAD, or a speech-to-speech API's speech events, report the end of speech
      </td>

      <td>
        On a timer, or from a fixed start, as in 

        [Choose when to send](#choose-when-to-send)
      </td>
    </tr>

    <tr>
      <td>
        **What a clip holds**
      </td>

      <td>
        Audio around detected speech, including short pauses and any background sound at the same time. Longer gaps between segments are left out.
      </td>

      <td>
        Everything on the line: speech, silence, ringing, beeps, hold music, background noise
      </td>
    </tr>

    <tr>
      <td>
        **Use it for**
      </td>

      <td>
        Questions about how someone speaks
      </td>

      <td>
        Questions about sounds other than speech, or about the line before anyone speaks
      </td>
    </tr>
  </tbody>
</table>

For speech segments:

1. Choose whose voice. For questions about how one speaker sounds, send only that speaker's audio, taken from their own input track before it is mixed with other voices, such as a voice agent's. For questions about the whole conversation, send all sides mixed into one clip.
2. Send at least 5 seconds where you can. Extend a shorter clip with the speaker's earlier speech from the same turn, never with silence. If the clip is still shorter than 1 second, skip the request and continue without an answer, since Hume Audio Decisions rejects audio under 1 second with `audio_too_short`. See the [live audio guidance](/audio-decisions/docs/preparing-your-audio#live-audio).
3. Do not join speech across another speaker's turns. A "Yes." in answer to a question carries its own tone. Joined to the speaker's previous sentence, the "Yes." tends to get the same answer as that sentence.

### Keep or skip silence

Pauses carry meaning: how long someone waits before answering, and where they stop mid-sentence. A speech segment keeps the pauses inside it but drops the gaps between segments, so pick the audio by what the question asks.

<table>
  <thead>
    <tr>
      <th>
        The question is about
      </th>

      <th>
        Pauses in the clip
      </th>

      <th>
        A clip that is mostly silence
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        How the speaker sounds, such as tone, pace, or hesitation
      </td>

      <td>
        Speech segments work. Pauses inside a segment keep their real length.
      </td>

      <td>
        Skip it. There is no voice to judge.
      </td>
    </tr>

    <tr>
      <td>
        Pauses themselves, such as "Does the speaker pause for half a second or more within their turn?"
      </td>

      <td>
        Send the continuous raw audio of the whole turn, cut on a timer or shortly after the speaker stops.
      </td>

      <td>
        Send it.
      </td>
    </tr>

    <tr>
      <td>
        Sounds other than speech, such as ringing or a beep
      </td>

      <td>
        Send the raw stream as it arrived.
      </td>

      <td>
        Send it.
      </td>
    </tr>
  </tbody>
</table>

## Choose when to send

**When you send decides how much of the call each answer covers and how soon you get it.** Pick the strategy that fits your question. Each clip can hold up to 5 minutes of audio.

<table>
  <thead>
    <tr>
      <th>
        Strategy
      </th>

      <th>
        A clip goes out
      </th>

      <th>
        Each answer covers
      </th>

      <th>
        Use it for
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        [End of speech](#send-at-the-end-of-speech)
      </td>

      <td>
        Each time VAD reports that the speaker stopped
      </td>

      <td>
        The latest speech, extended to 5 seconds within the turn
      </td>

      <td>
        How the speaker said something, such as "Does the caller sound frustrated?"
      </td>
    </tr>

    <tr>
      <td>
        [Fixed windows](#send-fixed-windows)
      </td>

      <td>
        On a timer, such as every 5 seconds
      </td>

      <td>
        The last window, from the raw stream or from speech segments
      </td>

      <td>
        Sounds on the line and checks that run for the whole call, such as "Is background noise making the speaker hard to hear?"
      </td>
    </tr>

    <tr>
      <td>
        [Expanding window](#send-an-expanding-window)
      </td>

      <td>
        At 1, 2, then 4 seconds from a fixed start, until the answer is clear
      </td>

      <td>
        Everything since the start
      </td>

      <td>
        One early decision, such as "Is this a voicemail greeting?"
      </td>
    </tr>

    <tr>
      <td>
        [Slices](#send-overlapping-slices)
      </td>

      <td>
        When a stretch of audio ends, or on a timer
      </td>

      <td>
        One short slice, compared with its neighbors
      </td>

      <td>
        Finding where an event happens, such as "Does someone laugh?"
      </td>
    </tr>
  </tbody>
</table>

### Send at the end of speech

Sending at the end of speech gets an answer about each thing the speaker says, often before the agent replies. Send a clip each time VAD reports that the speaker stopped. VAD ends a segment after its own stretch of silence, so the clip ends at a pause, though the speaker may continue the same sentence after the pause.

Clips cut at the speaker's pauses

Speaker Speech

Agent

Agent speaks

Clip 1 4.2 s

Clip 2 5 s

Clip 3 2.7 s

0 s 5 10 15 s

Example times. Each clip starts a little before the speech and ends a little after it. Clip 2 is the short second phrase extended to 5 seconds with the end of the first phrase. Of the 2-second pause between the phrases, Clip 2 keeps 0.2 s after the first and 1 s before the second, and leaves out the 0.8 s between them. Clip 3 follows the agent's turn, so it is sent on its own.

1. **Trigger:** the VAD's end of speech, or the speech-stopped event of a speech-to-speech API.
2. **Audio:** the latest speech, extended to 5 seconds within the turn. For speech longer than 5 minutes, send the latest 5 minutes.
3. **Requests:** one at a time per call. If the speaker pauses again while a request runs, hold only the newest clip and send it when the request finishes. Count the [wait limit](#set-the-wait-limit-in-code) from the newest end of speech.

### Send fixed windows

Your code cuts a window on a timer, whatever the speaker is doing, so fixed windows suit questions about the line itself, such as background noise or a second voice, and checks that run for the whole call.

5-second windows on the raw stream

Speaker Speech

Windows

1

2

3

Every 5 s

0 s 5 10 15 s

Window 2 starts in the middle of a sentence. Each window holds whatever sound was on the line in that stretch.

Window length and how often you send a window are separate choices. A 5-second window sent every second gives a new answer each second, each about the latest 5 seconds. Overlapping windows keep a fresh answer ready for a reply that doesn't wait for it, but they send five times as many requests.

5-second windows, one sent every second

Sent at 5 s

Sent at 6 s

Sent at 7 s

Sent at 8 s

Sent at 9 s

Sent at 10 s

0 s 5 10 s

Each window holds the latest 5 seconds and is sent as it ends, so each second of audio is sent in up to five windows.

1. **Trigger:** a timer, such as every 5 seconds, or every second for overlapping windows.
2. **Audio:** the last window, from the raw stream or from speech segments. A 5-second window of speech segments can hold less than 5 seconds, because gaps are left out, so extend it to 5 seconds before you send it.
3. **Requests:** one per window per call. If the previous request is still running when the next window is due, skip that window. After a failed request, continue with the next window, and follow `Retry-After` on a `429`. See [Plan requests and billing](#plan-requests-and-billing) for how many calls the rate limit covers.
4. **Example:** ask "Is background noise making the speaker hard to hear?" and show a connection warning when the answer passes your threshold on two windows in a row.

### Send an expanding window

An expanding window gets one decision as early as the audio allows, such as whether an outbound call reached a person or a voicemail greeting. Send 1 second from a fixed start, such as when the call connects, then 2 seconds from the same start, then 4.

Windows that grow from the start of the call, asked "Is this a voicemail greeting?"

Line Greeting

Request 1 0.41 next

Request 2 0.63 next

Request 3 0.94 passes

0 s 1 2 4 6 s

Example answers with an upper threshold of 0.9 and a lower one of 0.1. Each request goes out as soon as its audio is recorded, at 1, 2, and 4 seconds, and each answer arrives 0.22 s later, before the next window is ready. The answer to request 3 arrives at 4.22 seconds, between the dashed lines, and passes, so the agent sends no further requests and waits for the greeting to end before leaving its message.

1. **Start length:** the shortest clip that gives reliable answers on your own test calls. Voicemail detection needs an early decision, so this example starts at 1 second and doubles. Most questions need 5 seconds; each [use case](/audio-decisions/docs/use-cases) lists its minimum.
2. **Audio:** everything from the fixed start, doubling in length each time, up to a maximum of 5 minutes or less.
3. **Requests:** one at a time. To handle each answer:
   1. Set an upper and a lower threshold on your own calls.
   2. Above the upper threshold, stop sending and wait for the voicemail greeting to end, then leave your message.
   3. Below the lower threshold, start the conversation.
   4. Between them, send the next, doubled window.
   5. At the maximum length, or when a request fails, use your fallback, such as treating the call as a person.

### Send overlapping slices

Slices find where in the audio an event happens, such as a laugh. They use the same overlapping windows as [fixed windows](#send-fixed-windows), and you compare their answers:

1. Cut the audio into short overlapping slices.
2. Ask every slice the same question.
3. Compare each answer with a threshold you set on your own calls. A passing slice marks a likely event somewhere inside that slice.
4. Merge neighboring passing slices into one detected stretch.
5. Check your detections against labeled recordings.

2-second slices that start every second, asked "Does someone laugh?"

Speaker Laugh

0 to 2 s 0.04

1 to 3 s 0.46

2 to 4 s 0.93

3 to 5 s 0.51

4 to 6 s 0.07

0 s 2 4 6 s

Example answers with a threshold of 0.7. The laugh lasts 0.8 seconds, so 2-second slices every second hold it whole in at least one. Only the slice from 2 to 4 seconds passes, at 0.93, so the laugh is likely inside that slice.

1. **Trigger:** the end of the stretch you want to examine, or a timer.
2. **Audio:** make each slice at least twice as long as the longest event you expect, and start a new slice at least every half slice. With that spacing, every event of that length fits whole in one slice. Use slices shorter than 5 seconds only for questions that give reliable answers at that length, such as whether someone laughs.
3. **Requests:** send the slices one after another, oldest first. Each counts toward the [rate limit](/audio-decisions/docs/limits#rate-limits).

## Voice agents

**In a voice agent, answers from Hume Audio Decisions can change the agent's next reply or trigger an action in your backend, without replacing any part of your pipeline.**

### Where Hume Audio Decisions fits

#### In a cascade

Speech-to-text, an LLM, and text-to-speech run one after another, and Hume Audio Decisions runs beside them. A request sent when VAD reports the end of speech runs while turn detection decides whether the turn is over, so an answer the reply waits for often arrives before the LLM would start.

A cascade with Hume Audio Decisions

<defs>
  <path d="M1 1 L9 5 L1 9" />
</defs>

<g>
  <rect x="4" y="20" width="112" height="44" rx="8" />

  Caller audio
</g>

<g>
  <rect x="150" y="2" width="112" height="34" rx="8" />

  Speech-to-text
</g>

<g>
  <rect x="150" y="48" width="112" height="34" rx="8" />

  VAD
</g>

<g>
  <rect x="290" y="48" width="112" height="34" rx="8" />

  Turn detection
</g>

<g>
  <rect x="430" y="20" width="84" height="44" rx="8" />

  LLM
</g>

<g>
  <rect x="530" y="20" width="112" height="44" rx="8" />

  Text-to-speech
</g>

<g>
  <rect x="658" y="20" width="98" height="44" rx="8" />

  Agent audio
</g>

<path d="M402 65 H416 V42" />

<g>
  <rect x="150" y="146" width="200" height="48" rx="8" />

  Hume Audio Decisions
</g>

<text x="68" y="118">
  raw stream
</text>

<text x="212" y="118">
  speech segments
</text>

<text x="478" y="118">
  inline: the reply waits
</text>

<g>
  <rect x="560" y="156" width="196" height="52" rx="8" />

  Your backend

  actions, alerts, later turns
</g>

background: no wait

The solid arrow from Hume Audio Decisions into the LLM is the inline path: the answer reaches the LLM before it writes the reply. The dashed arrow is the background path: the answer goes to your backend without holding the reply. An agent can use both paths by sending one request with all its questions, then using some answers inline and passing the rest to your backend.

#### With a speech-to-speech model

One model takes the caller's audio and returns the reply as audio, so there is no transcript step for the request to run beside.

A speech-to-speech model with Hume Audio Decisions

<defs>
  <path d="M1 1 L9 5 L1 9" />
</defs>

<g>
  <rect x="4" y="20" width="112" height="44" rx="8" />

  Caller audio
</g>

<g>
  <rect x="200" y="20" width="280" height="44" rx="8" />

  Speech-to-speech model
</g>

<g>
  <rect x="644" y="20" width="112" height="44" rx="8" />

  Agent audio
</g>

<g>
  <rect x="150" y="146" width="200" height="48" rx="8" />

  Hume Audio Decisions
</g>

<text x="68" y="112">
  raw stream or speech segments
</text>

<text x="436" y="112">
  inline: the reply waits
</text>

<g>
  <rect x="560" y="156" width="196" height="52" rx="8" />

  Your backend

  actions, alerts, later turns
</g>

background: no wait

The solid arrow from Hume Audio Decisions into the model is the inline path: the model's response waits until the answer is added. The dashed arrow is the background path: the answer goes to your backend or into a later response.

* The model's own speech events can cut speech segments without a VAD of your own. The OpenAI Realtime API, for example, sends `input_audio_buffer.speech_started` and `input_audio_buffer.speech_stopped`.
* To use an answer on the current response, hold the response until the answer arrives, as in [Inline](#inline).
* A background answer goes into the conversation for a later response, or to your backend.

### Choose whether the reply waits

A reply that waits can respond to how the caller sounded on this turn. A reply that doesn't wait adds no delay and uses the answer later. Turn detection, speech-to-text, the LLM, and text-to-speech all run between the caller stopping and the agent speaking. Inline and background requests both run alongside these steps. An inline request also holds the LLM until the answer arrives.

**Inline: the reply waits for the answer**

1. The caller speaks and stops at 3 seconds.
2. VAD reports the end of speech at 3.3 seconds, and your code sends the clip to Hume Audio Decisions.
3. While turn detection decides whether the turn is over, the answer arrives, at 3.52 seconds.
4. Turn detection ends the turn at 3.8 seconds. The answer is already in, so the reply starts on time and responds to how the caller sounded.

Example times. VAD waits 0.3 s of silence before it reports the end of speech, and turn detection takes 0.5 s here; yours may differ. The response time, between the dashed lines, is 0.22 s. If an answer arrives after turn detection, the reply waits for it, up to the wait limit.

**Background: the reply starts without waiting**

1. The caller speaks and stops at 3 seconds.
2. VAD reports the end of speech at 3.3 seconds, and your code sends the clip to Hume Audio Decisions.
3. The answer arrives at 3.52 seconds and goes to your backend, such as an alert to a supervisor.
4. Turn detection ends the turn at 3.8 seconds, and the reply starts without waiting. An answer that arrives later reaches the next turn.

Example times. VAD waits 0.3 s of silence before it reports the end of speech, and turn detection takes 0.5 s here; yours may differ. The response time, between the dashed lines, is 0.22 s.

<table>
  <thead>
    <tr>
      <th />

      <th>
        Inline
      </th>

      <th>
        Background
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **The LLM waits for the answer**
      </td>

      <td>
        Yes
      </td>

      <td>
        No
      </td>
    </tr>

    <tr>
      <td>
        **The answer is used**
      </td>

      <td>
        On the reply to the same speech
      </td>

      <td>
        On the current reply if it has already arrived when the LLM starts, otherwise on a later turn or in your backend
      </td>
    </tr>

    <tr>
      <td>
        **Use it for**
      </td>

      <td>
        Adapting the reply to how the caller sounded, deciding whether to start the conversation
      </td>

      <td>
        Tool calls and other actions in your backend, alerts, moderation, context for later turns, call notes
      </td>
    </tr>
  </tbody>
</table>

#### Inline

The LLM writes its reply once it has both the transcript and the answer about the same speech, so the reply can respond to how the caller sounded, such as offering to slow down when "I'm fine" sounds sad. Pipecat's `PerceptionReader` works this way.

1. Send the clip as soon as VAD reports the end of speech. See [Choose when to send](#choose-when-to-send).
2. Hold the LLM until the answer arrives, up to a wait limit. Pipecat's default is 2 seconds after the caller stops. The response time is 0.22 seconds, so the limit takes effect only when a request fails or the network is slow. If the limit passes, reply without the answer.
3. In a cascade, put the answer at the head of the caller's message to the LLM, such as `[tone sad 0.81] I'm fine. Let's just get the new booking done.`, and say what it means in the system prompt, such as `tone: the likeliest tones, each with the estimated share of raters who would choose it`.
4. With a speech-to-speech model, hold the model's response yourself. In the OpenAI Realtime API, for example:

   1. Set `create_response` to `false` in `turn_detection`, so the model does not respond on its own at the end of the turn.
   2. When the answer arrives, send `conversation.item.create` with a message holding the answer line.
   3. Send `response.create` to start the response.
   4. At the wait limit, send `response.create` without the answer.

   Without the hold, the model starts replying at the speech-stopped event, so holding the reply adds the full response time.

#### Background

A background request adds no time to the reply.

1. Send the clip as soon as VAD reports the end of speech, and start the LLM without waiting.
2. Tag each request with the turn it came from. When the answer arrives, add it to the context with that turn, such as `On turn 12, the caller sounded frustrated (0.82).`, and drop answers from turns older than the last one or two.
3. To act in your backend instead, call a tool, update a record, hand the call to a person, raise an alert past a threshold, or save the answer with the call.

#### Background answers on the current reply

A background answer reaches the current reply when it arrives before the LLM starts. Send [overlapping windows](#send-fixed-windows) while the caller speaks, such as the latest 5 seconds every second.

1. When the turn ends, add the newest answer that has already arrived to the caller's message. The reply does not wait.
2. That answer's window ends before the turn does, so it can leave out the caller's last words.
3. Overlapping windows send five times the requests of back-to-back windows. See [Plan requests and billing](#plan-requests-and-billing).

#### Set the wait limit in code

An inline reply waits for the request, so give it a wait limit:

1. Count the limit from when the caller stopped. VAD reports the end of speech after its own stretch of silence, such as Silero's `stop_secs`, so subtract that silence from the event's timestamp, as Pipecat does.
2. Send the request once. By default, the [Python](/audio-decisions/docs/quickstart/python) and [TypeScript](/audio-decisions/docs/quickstart/typescript) SDKs wait up to 60 seconds and retry a failed request twice.
3. The Python and TypeScript examples stop waiting 2 seconds after the caller stops. The cURL example allows 2 seconds from when the request goes out; to hold it to the same limit, work out the time left in your application and pass that to `--max-time`.

#### cURL

```bash
curl https://api.cloud.hume.ai/v0/decisions/file \
  --max-time 2 \
  -H "X-Hume-Api-Key: $HUME_API_KEY" \
  -F 'request={
    "questions": {
      "hesitant": {
        "type": "noul",
        "instructions": "Does the caller sound hesitant or unsure?"
      }
    }
  };type=application/json' \
  -F file=@clip.wav
```

#### Python

```python
import asyncio
import time

import httpx
from hume_audio_decisions import (
    AsyncDecisionsClient,
    DecisionsRequest,
    Question_Noul,
)
from hume_audio_decisions.core import ApiError

WAIT_LIMIT = 2.0  # Seconds after the caller stops.

client = AsyncDecisionsClient()

request = DecisionsRequest(
    questions={
        "hesitant": Question_Noul(
            instructions="Does the caller sound hesitant or unsure?",
        ),
    },
)

async def ask(clip: bytes, stopped_at: float):
    # stopped_at: time.monotonic() at VAD's end-of-speech event,
    # minus the silence VAD waits for before it reports the end.
    remaining = stopped_at + WAIT_LIMIT - time.monotonic()
    if remaining <= 0:
        return None
    try:
        return await asyncio.wait_for(
            client.decisions.create(
                file=("clip.wav", clip, "audio/wav"),
                request=request,
                request_options={"max_retries": 0},
            ),
            timeout=remaining,
        )
    except (ApiError, httpx.HTTPError, asyncio.TimeoutError):
        return None  # Reply without the answer.
```

#### TypeScript

```typescript
import { DecisionsClient } from "@humeai/audio-decisions";

const WAIT_LIMIT_MS = 2000; // After the caller stops.

const client = new DecisionsClient();

// stoppedAt: performance.now() at VAD's end-of-speech event,
// minus the silence VAD waits for before it reports the end.
async function ask(clip: Uint8Array, stoppedAt: number) {
  const remainingMs = Math.floor(
    stoppedAt + WAIT_LIMIT_MS - performance.now(),
  );
  if (remainingMs <= 0) return undefined;
  try {
    return await client.decisions.create(
      {
        file: {
          data: clip,
          filename: "clip.wav",
          contentType: "audio/wav",
        },
        request: {
          questions: {
            hesitant: {
              type: "noul",
              instructions: "Does the caller sound hesitant or unsure?",
            },
          },
        },
      },
      { abortSignal: AbortSignal.timeout(remainingMs), maxRetries: 0 },
    );
  } catch {
    // A DecisionsError, including the abort at the wait limit.
    return undefined; // Reply without the answer.
  }
}
```

1. A request stopped at the wait limit returns no answer. To use a late answer on a later turn instead, release the reply at the limit but let the request run on in its own task, and add its answer when it arrives.
2. For requests sent on a timer or in an expanding window, count the wait limit from each send, and also set a maximum listening time, after which you use your fallback, as in [Send an expanding window](#send-an-expanding-window).

### Common setups

<table>
  <thead>
    <tr>
      <th>
        Goal
      </th>

      <th>
        Reply
      </th>

      <th>
        Audio
      </th>

      <th>
        When to send
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        Adapt the reply to how the caller sounded
      </td>

      <td>
        Inline
      </td>

      <td>
        Speech segments
      </td>

      <td>
        End of speech
      </td>
    </tr>

    <tr>
      <td>
        Adapt a speech-to-speech model's reply
      </td>

      <td>
        Inline
      </td>

      <td>
        Speech segments, cut at the model's speech events
      </td>

      <td>
        End of speech
      </td>
    </tr>

    <tr>
      <td>
        Tell a voicemail greeting from a person on an outbound call
      </td>

      <td>
        Inline: the agent's first words wait for the answer
      </td>

      <td>
        Raw stream
      </td>

      <td>
        Expanding window
      </td>
    </tr>

    <tr>
      <td>
        Alert a supervisor or call a tool when a caller sounds upset
      </td>

      <td>
        Background
      </td>

      <td>
        Speech segments
      </td>

      <td>
        End of speech
      </td>
    </tr>

    <tr>
      <td>
        Notice background noise or a second voice on the line
      </td>

      <td>
        Background
      </td>

      <td>
        Raw stream
      </td>

      <td>
        Fixed windows
      </td>
    </tr>

    <tr>
      <td>
        Find when the caller laughs
      </td>

      <td>
        Background
      </td>

      <td>
        Raw stream
      </td>

      <td>
        Slices
      </td>
    </tr>
  </tbody>
</table>

### Checklist for other frameworks

[Pipecat](/audio-decisions/docs/integrations/pipecat) handles these steps for you. In any other framework, or an agent you build yourself, your code needs to:

<table>
  <thead>
    <tr>
      <th>
        Step
      </th>

      <th>
        How
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        Copy the caller's audio
      </td>

      <td>
        Copy each frame where the agent receives it and pass the frame on unchanged, so speech-to-text gets it without delay.
      </td>
    </tr>

    <tr>
      <td>
        Cut a clip
      </td>

      <td>
        Keep a rolling buffer of the incoming audio. At the trigger you chose in 

        [Choose when to send](#choose-when-to-send)

        , cut the clip and convert it to 16 kHz, 16-bit mono WAV. Hume Audio Decisions processes audio at 16 kHz, so a higher rate only makes the upload larger; a 5-second clip is about 160 KB. For speech segments, take the latest speech with up to 1 second before it and extend it to 5 seconds with earlier speech from the same turn. For the raw stream, keep the whole interval, including silence and other sounds.
      </td>
    </tr>

    <tr>
      <td>
        Clear the buffer
      </td>

      <td>
        For speech segments, drop earlier speech when the agent's turn ends. On a mute, drop it and stop collecting, so no clip joins speech from before and after the mute.
      </td>
    </tr>

    <tr>
      <td>
        Send the request
      </td>

      <td>
        From a background task, so audio keeps flowing while the request runs, with one client reused for every clip so requests share an open connection, one request at a time per call, with a 

        [wait limit](#set-the-wait-limit-in-code)

         on inline requests.
      </td>
    </tr>

    <tr>
      <td>
        Use the answers inline
      </td>

      <td>
        Hold the LLM until the answer arrives or the wait limit passes, then add the answer to the caller's message, or reply without it.
      </td>
    </tr>

    <tr>
      <td>
        Use the answers in the background
      </td>

      <td>
        Start the LLM as usual, with any answer that has already arrived. Add later answers to the context for the next turn, or send them to your backend.
      </td>
    </tr>

    <tr>
      <td>
        Keep late answers
      </td>

      <td>
        Release the reply at the wait limit but let the request run on in its own task, so its answer can still reach a later turn.
      </td>
    </tr>
  </tbody>
</table>

## Other live audio

**Outside a voice agent, answers feed a dashboard, an alert, or on-screen feedback, so nothing waits for them: send each clip as it is cut and act on each answer as it arrives.**

<table>
  <thead>
    <tr>
      <th>
        Audio
      </th>

      <th>
        What to ask
      </th>

      <th>
        How to cut the clips
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        A room, a device, or a broadcast
      </td>

      <td>
        Whether a named sound is heard, such as an alarm or running water
      </td>

      <td>
        [Slices](#send-overlapping-slices)

         of the raw stream, so each sound fits whole in at least one
      </td>
    </tr>

    <tr>
      <td>
        Calls between people
      </td>

      <td>
        How each person sounds, with the person named in the question, so a supervisor can see which calls need attention
      </td>

      <td>
        [Fixed windows](#send-fixed-windows)

        , with both sides mixed into one clip
      </td>
    </tr>

    <tr>
      <td>
        Speaking practice
      </td>

      <td>
        Speaking rate or filler words, shown to the speaker while they talk
      </td>

      <td>
        Fixed 5-second windows from the microphone, as in the 

        [Answering live audio](/audio-decisions/cookbooks/answering-live-audio)

         cookbook
      </td>
    </tr>

    <tr>
      <td>
        Call and stream quality
      </td>

      <td>
        Whether the audio cuts out or there is noise behind the speaker
      </td>

      <td>
        Fixed windows of the audio as the listener receives it
      </td>
    </tr>
  </tbody>
</table>

> **Info**
>
> When every clip needs an answer, as in the [Answering live audio](/audio-decisions/cookbooks/answering-live-audio) app, requests overlap, so count them against the rate limit, and keep each answer with its own clip, since answers can arrive out of order.

## Choose questions

1. Ask about what a listener hears in the clip. "Does the caller sound hesitant or unsure?" and "Are several people talking?" are answered from the audio. "Who is this caller?" is not.
2. Keep inline questions few. Each answer adds a line to every caller message the LLM reads. Start with one [`noul`](/audio-decisions/docs/primitives#noul) question, such as "Does the caller sound frustrated?", and add more once you have seen its answers on your own calls.
3. Send the same questions, in the same order, with every clip. All the questions in a request are answered together, so an answer can change when a question is added, removed, or moved. Keeping the questions fixed keeps answers comparable across turns. See [Writing questions](/audio-decisions/docs/writing-questions).
4. Describe the call in `context`. For example, `An outbound call from an airline's rebooking agent. The caller is the customer.` A transcript of the turn can go on its own line, labeled `Transcript:`. With Pipecat, `PerceptionReader` always sets `context` to the bot's last reply, so you cannot add your own description; see [Pipecat](/audio-decisions/docs/integrations/pipecat#how-it-sends-audio).

## Act on answers

Hume Audio Decisions answers each clip on its own and doesn't combine answers across clips, so your code does.

1. Tell the agent how to use each answer. Report a [`noul`](/audio-decisions/docs/primitives#noul) answer as its value and a [`choice`](/audio-decisions/docs/primitives#choice) answer as its likeliest options, and say what to do past a threshold, such as `If hesitation is 0.7 or above, check that the caller is following.`
2. Set thresholds on clips cut the way you will cut them live. The same question gives a different [value](/audio-decisions/docs/interpreting-answers) on one turn than on a whole call, so record and label clips from your own calls, then set each threshold as in [Setting thresholds](/audio-decisions/docs/interpreting-answers#setting-thresholds).
3. For a decision about the whole call, require the threshold to pass on two or three clips in a row. One clip covers a few seconds, so wait for a run of passing answers before handing a call to a person, for example.
4. Store each answer as it arrives, with its `run_id` and the start and end of its clip in the call. Hume Audio Decisions returns each answer only once, in its response. See [Storing answers](/audio-decisions/docs/interpreting-answers#storing-answers).

## Plan requests and billing

Every API key in an organization shares a limit of [600 requests per minute](/audio-decisions/docs/limits#rate-limits), across every call at once. Each request is [billed](/audio-decisions/docs/billing) by its input tokens, which come mostly from its audio, so audio you send in two clips is billed twice. [Estimate input tokens](/audio-decisions/reference/decisions/input-tokens) takes an audio duration from 1 second to 5 minutes and your questions, and returns the count before you send any audio.

<table>
  <thead>
    <tr>
      <th>
        Schedule
      </th>

      <th>
        Requests per call per minute
      </th>

      <th>
        Calls that reach 600 per minute
      </th>

      <th>
        How often the same audio is sent
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        End of speech, at 6 pauses a minute
      </td>

      <td>
        6
      </td>

      <td>
        100
      </td>

      <td>
        Once, plus earlier speech re-sent to extend a short clip
      </td>
    </tr>

    <tr>
      <td>
        5-second windows, one after another
      </td>

      <td>
        12
      </td>

      <td>
        50
      </td>

      <td>
        Once
      </td>
    </tr>

    <tr>
      <td>
        5-second windows, one sent every second
      </td>

      <td>
        60
      </td>

      <td>
        10
      </td>

      <td>
        Up to five times
      </td>
    </tr>
  </tbody>
</table>

## Next steps

#### [Answering live audio](/audio-decisions/cookbooks/answering-live-audio)

Run a Next.js app that sends microphone audio in 5-second clips.

#### [Limits](/audio-decisions/docs/limits)

Check audio length, question count, and request rate before you scale up.

#### [Interpreting answers](/audio-decisions/docs/interpreting-answers)

Turn values into actions with thresholds set on your own calls.