> This page is for Audio Decisions.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://dev.hume.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://dev.hume.ai/_mcp/server.

# Evaluating many recordings

**Upload a set of recordings, get answers to the same questions about each one and compare them row by row.** Clone this Gradio app to follow along, or fork it as a starting point.

![The app after a run over eight recordings, with the recordings and the request in the sidebar. The table has one row per part with its billed tokens and one column per question, and the selected part's audio is beside its full response.](/_fern-files/_img/421851107819d528713aac4b9460ac03b36920e6fbf6747014b819da0cd6840a.webp)![The app after a run over eight recordings, with the recordings and the request in the sidebar. The table has one row per part with its billed tokens and one column per question, and the selected part's audio is beside its full response.](/_fern-files/_img/878e2c10076d2bc40e7af3fc845e1dd692339ca9da6a4f086505b4be756c2c01.webp)

By the end you will have the app running locally, and you will know how it stays within the rate limit, sends long recordings in parts, estimates a run's tokens, picks up a stopped run and shows the answers.

<table>
  <tbody>
    <tr>
      <th width="20%">
        Repository
      </th>

      <td>
        [hume-audio-decisions-gradio-uploader-example](https://github.com/HumeAI/hume-audio-decisions-gradio-uploader-example)

         on GitHub
      </td>
    </tr>

    <tr>
      <th>
        SDK
      </th>

      <td>
        [`hume-audio-decisions`](https://pypi.org/project/hume-audio-decisions/)

         on PyPI
      </td>
    </tr>

    <tr>
      <th>
        Python
      </th>

      <td>
        3.11 or later
      </td>
    </tr>
  </tbody>
</table>

## Prerequisites

1. Python 3.11 or later, and git.
2. An API key from the [API keys page](https://cloud.hume.ai/settings/organization/api-keys) of the Hume Platform. See [Authentication](/audio-decisions/docs/authentication).
3. Recordings in MP3, or in WAV with PCM or floating-point samples, each at least 5 seconds long. The app sends a recording longer than 5 minutes in parts.

## Run the example

Clone the repository and install its dependencies. If you use uv, it needs version 0.13.6 or later and installs the Python version in `.python-version` for you.

#### uv

#### Shell

```bash
git clone https://github.com/HumeAI/hume-audio-decisions-gradio-uploader-example.git
cd hume-audio-decisions-gradio-uploader-example
uv sync
```

#### pip

#### Shell

```bash
git clone https://github.com/HumeAI/hume-audio-decisions-gradio-uploader-example.git
cd hume-audio-decisions-gradio-uploader-example
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

Set your API key. The app creates the SDK client when it starts, which reads `HUME_API_KEY` on the server, so the key never reaches the browser. If the variable is not set, the app exits when it starts.

#### Shell

```bash
export HUME_API_KEY=your_api_key
```

Start the app and open [http://localhost:7860](http://localhost:7860).

#### uv

#### Shell

```bash
uv run app.py
```

#### pip

#### Shell

```bash
python app.py
```

Add recordings under **Recordings** in the sidebar, then press **Run**. To ask your own questions, edit the **Request** first. Each part of each recording gets a row with the input tokens it was billed and one column per question. Select a row to hear its part and see its full response. Press **Estimate tokens** before a run to see what it would be billed. During a run, **Stop** cancels the requests in flight, and pressing **Run** again sends only what has no answer yet. Results last as long as the browser tab, so download them as CSV or JSON before you reload the page.

## How it works

#### Parse the request

The **Request** editor starts with `DEFAULT_REQUEST`, which asks one question of each type: `noul`, `choice`, `score` and `span`. It is the `request` part of an API call as JSON, so a request from these docs pastes in unchanged.

**`questions.py (lines 1-21)`**

```python questions.py (lines 1-21)
DEFAULT_REQUEST = json.dumps(
    {
        "context": (
            "A recording of one or more people speaking, such as a phone call, "
            "a meeting or a voice note."
        ),
        "questions": {
            "background_noise": {
                "type": "noul",
                "instructions": "Is there distracting background noise?",
            },
            # ...
            "laughter": {
                "type": "span",
                "instructions": "When does someone laugh?",
            },
        },
    },
    indent=2,
    ensure_ascii=False,
)
```

`parse_request` checks the editor's text with the SDK's `DecisionsRequest` model before anything is sent, and raises a `ValueError` whose message the app shows on the **Request** editor. `json.loads` keeps only the last value of a repeated key, so `_reject_repeated_keys` rejects a repeated question ID that would otherwise drop a question.

**`questions.py (lines 24-49)`**

```python questions.py (lines 24-49)
def parse_request(text: str) -> DecisionsRequest:
    try:
        data = json.loads(text, object_pairs_hook=_reject_repeated_keys)
    except json.JSONDecodeError as error:
        position = f"line {error.lineno}, column {error.colno}"
        message = f"The request isn't valid JSON: {error.msg} at {position}."
        raise ValueError(message) from error
    # ...

    try:
        request = DecisionsRequest.model_validate(data)
    except ValidationError as error:
        raise ValueError(_describe(error)) from error
    # ...
    return request


def _reject_repeated_keys(pairs: list[tuple[str, object]]) -> dict[str, object]:
    # json.loads keeps only the last value of a repeated key, so a repeated
    # question ID would drop a question without a word.
    data: dict[str, object] = {}
    for key, value in pairs:
        if key in data:
            raise ValueError(f'The request repeats the key "{key}".')
        data[key] = value
    return data
```

#### Pace every request

`make_client` builds an `AsyncDecisionsClient` on the SDK's `DefaultAsyncHttpxClient` with a request hook, so every request the SDK sends waits for its turn, retries included. `REQUESTS_PER_MINUTE` is 50, which leaves room for other work under the limit of 60 that every API key in your organization shares. The SDK retries a rate limit or server error up to `MAX_RETRIES` times before it reports the error. See [Limits](/audio-decisions/docs/limits#rate-limits).

**`evaluation.py (lines 1-7)`**

```python evaluation.py (lines 1-7)
# Every API key in an organization shares a limit of 60 requests a minute, so this
# leaves room for other work.
REQUESTS_PER_MINUTE = 50
CONCURRENT_RECORDINGS = 4
MIN_PART_MS = 5_000
MAX_PART_MS = 300_000
MAX_RETRIES = 5
```

**`evaluation.py (lines 28-37)`**

```python evaluation.py (lines 28-37)
def make_client(
    *,
    requests_per_minute: float = REQUESTS_PER_MINUTE,
    transport: httpx.AsyncBaseTransport | None = None,
) -> AsyncDecisionsClient:
    pacer = _Pacer(interval=60 / requests_per_minute)
    http_client = DefaultAsyncHttpxClient(
        event_hooks={"request": [pacer]}, transport=transport
    )
    return AsyncDecisionsClient(httpx_client=http_client, max_retries=MAX_RETRIES)
```

`_Pacer` starts requests at least 1.2 seconds apart. Each request claims its start time before it sleeps, so requests that wait at the same time queue up in order without a lock.

**`evaluation.py (lines 42-53)`**

```python evaluation.py (lines 42-53)
class _Pacer:
    def __init__(self, interval: float) -> None:
        self._interval = interval
        self._next_start = time.monotonic()

    async def __call__(self, _request: httpx.Request) -> None:
        now = time.monotonic()
        start = max(now, self._next_start)
        # Each request claims its start before sleeping, so requests that wait at
        # the same time queue up without a lock.
        self._next_start = start + self._interval
        await asyncio.sleep(start - now)
```

`_run_workers` starts `CONCURRENT_RECORDINGS` workers that share one iterator, so four recordings are sent at a time and each goes to only one worker. `main` in `app.py` creates one client for the whole app, so every browser tab shares the same pacer.

**`evaluation.py (lines 58-69)`**

```python evaluation.py (lines 58-69)
async def _run_workers(
    work: Callable[[Stretch, Callable[[_T], None]], Awaitable[None]],
    stretches: Iterable[Stretch],
    stop: asyncio.Event,
) -> AsyncIterator[_T]:
    updates: asyncio.Queue[_T | None] = asyncio.Queue()
    # The workers share one iterator, so each stretch goes to only one of them.
    remaining = iter(stretches)

    async def worker() -> None:
        for stretch in remaining:
            await work(stretch, updates.put_nowait)
```

#### Send each recording in parts

`iter_create_chunked` cuts each recording into the fewest equal parts that fit the API's limits, none longer than `MAX_PART_MS`. Each part goes in its own request with the same `request`, and its result arrives as soon as that request is done, so the table fills in during the run. `start_ms` and `end_ms` let a run start partway through a recording. See Long recordings in the [Python SDK README](https://github.com/HumeAI/hume-audio-decisions-python-sdk#long-recordings).

**`evaluation.py (lines 74-91)`**

```python evaluation.py (lines 74-91)
async def _send_stretch(
    client: AsyncDecisionsClient,
    request: DecisionsRequest,
    stretch: Stretch,
    report: Callable[[Progress], None],
) -> None:
    report(Started(stretch))
    file = await asyncio.to_thread(stretch.path.open, "rb")
    with file:
        # The SDK leaves client.decisions untyped, so this annotation is what lets
        # mypy check the parts.
        parts: AsyncIterator[ChunkResult] = client.decisions.iter_create_chunked(
            request=request,
            file=(stretch.path.name, file),
            max_chunk_ms=MAX_PART_MS,
            start_ms=stretch.start_ms,
            end_ms=stretch.end_ms,
        )
```

A part whose request fails does not raise. Its error is on the part's result, and the table shows why it failed. Most errors also end that recording, so its later parts show as **Not sent**. An error from the API key or the request, such as a rejected key or an invalid question, would fail every other recording the same way, so it stops the whole run. See [Errors](/audio-decisions/docs/errors).

**`evaluation.py (lines 9-12)`**

```python evaluation.py (lines 9-12)
# These errors come from the key or the request, which every recording shares, so
# every other recording would fail the same way.
_BATCH_STOPPING_STATUSES = frozenset({401, 402, 403})
_BATCH_STOPPING_CODES = frozenset({"message_invalid", "question_invalid"})
```

**`evaluation.py (lines 92-99)`**

```python evaluation.py (lines 92-99)
        try:
            async for part in parts:
                report(PartResult(stretch, part))
                if _is_api_error(
                    part.error, _BATCH_STOPPING_STATUSES, _BATCH_STOPPING_CODES
                ):
                    report(BatchStopped(part.error))
                    raise _HaltError
```

The SDK raises `ValueError` only before it sends anything, when it can't read or cut the file, so the recording is shown as unreadable and nothing is billed for it.

**`evaluation.py (lines 100-105)`**

```python evaluation.py (lines 100-105)
        # The SDK raises ValueError only before it sends anything, when it can't
        # split the file.
        except ValueError as error:
            report(Unreadable(stretch, error))
        else:
            report(Finished(stretch))
```

#### Estimate the input tokens

**Estimate tokens** cuts each recording into the parts Run would send, with the SDK's `iter_audio_chunks` and the same settings. A part under 5 seconds is counted apart rather than estimated, since the API rejects it.

**`evaluation.py (lines 108-124)`**

```python evaluation.py (lines 108-124)
async def _estimate_stretch(
    client: AsyncDecisionsClient,
    request: DecisionsRequest,
    stretch: Stretch,
    report: Callable[[EstimateProgress], None],
) -> None:
    try:
        durations = await asyncio.to_thread(_durations, stretch)
    except ValueError as error:
        report(Unreadable(stretch, error))
        return
    for duration_ms in durations:
        # The API rejects a part this short, so Run's request for it would fail
        # without being billed.
        if duration_ms < MIN_PART_MS:
            report(TooShort(stretch, duration_ms))
            continue
```

`input_tokens` takes each part's duration with the request's `questions` and `context`, and no audio is sent. A total that leaves out parts would understate the run, so any error stops the estimate. The estimate can differ slightly from the input tokens each response is billed, `received.input_tokens`. See [Estimate input tokens](/audio-decisions/reference/decisions/input-tokens).

**`evaluation.py (lines 125-138)`**

```python evaluation.py (lines 125-138)
        try:
            # The SDK leaves client.decisions untyped, so this annotation is what
            # lets mypy check the response.
            response: InputTokensResponse = await client.decisions.input_tokens(
                audio_duration_ms=duration_ms,
                questions=request.questions,
                context=request.context,
            )
        # The SDK has already retried what can pass, and a total that leaves out
        # parts would understate the run, so any error stops the estimate.
        except (ApiError, httpx.HTTPError) as error:
            report(BatchStopped(error))
            raise _HaltError from error
        report(Estimated(stretch, duration_ms, response.estimated_input_tokens))
```

#### Pick up where a run stopped

An answer can change when other questions are added, removed or reordered, so `_same` compares the order of the questions as well as the request. While the table holds answers, Run refuses a different request until you press **Clear results**.

**`results.py (lines 1-8)`**

```python results.py (lines 1-8)
def _same(request: DecisionsRequest, other: DecisionsRequest | None) -> bool:
    # DecisionsRequest equality ignores the order of the questions, but the model
    # answers them together, so a reordered request can answer differently.
    return (
        other is not None
        and request == other
        and list(request.questions) == list(other.questions)
    )
```

When you press **Run** again, `_resend_from` finds where each recording starts again: at the first part that failed for a reason that can pass, such as a timeout, or after the last part that was sent. A recording that stopped at a part whose file the API rejected is not sent again, because that part would fail the same way, and a recording with every answer is skipped.

**`results.py (lines 11-27)`**

```python results.py (lines 11-27)
def _resend_from(recording: _Recording) -> int | None:
    if recording.unreadable is not None:
        return None
    start_ms = 0
    for part in recording.parts:
        if _will_resend(part):
            return start_ms
        if part.status == "not_attempted":
            return None
        # The next part starts where this one ends, not at its own start_ms, because
        # an MP3 part's start_ms includes frames repeated from before it.
        start_ms = part.end_ms
    return None if recording.finished else start_ms


def _will_resend(part: ChunkResult) -> bool:
    return part.error is not None and not is_file_error(part.error)
```

#### Show the answers

Each cell is short enough to compare down a column: the share of raters who would answer yes to a `noul` question, the mean `score` on a scale from 0 to the highest level and the `choice` option with the largest share of raters, followed by that share. See [Interpreting answers](/audio-decisions/docs/interpreting-answers).

**`results.py (lines 32-39)`**

```python results.py (lines 32-39)
def _cell(answer: Answer, start_ms: int) -> str:
    match answer:
        case Answer_Noul():
            return f"{answer.noul:.2f}"
        case Answer_Choice():
            return f"{answer.choice} {answer.probabilities[answer.choice]:.2f}"
        case Answer_Score():
            return f"{answer.score:.2f} of {len(answer.legend) - 1}"
```

A `span` answer's times count from the start of its part, so `_cell` adds the part's start to place each span in the recording. Selecting a row cuts the part out again with `cut` in `evaluation.py`, so the player plays the audio that was sent, and its times, like the response's, count from the start of the part.

**`results.py (lines 40-42)`**

```python results.py (lines 40-42)
        case Answer_Span():
            times = (_clock(start_ms + m.audio_start_ms) for m in answer.spans)
            return ", ".join(times) or "none"
```

#### Complete files

**`questions.py`**

```python title="questions.py"
DEFAULT_REQUEST = json.dumps(
    {
        "context": (
            "A recording of one or more people speaking, such as a phone call, "
            "a meeting or a voice note."
        ),
        "questions": {
            "background_noise": {
                "type": "noul",
                "instructions": "Is there distracting background noise?",
            },
            # ...
            "laughter": {
                "type": "span",
                "instructions": "When does someone laugh?",
            },
        },
    },
    indent=2,
    ensure_ascii=False,
)


def parse_request(text: str) -> DecisionsRequest:
    try:
        data = json.loads(text, object_pairs_hook=_reject_repeated_keys)
    except json.JSONDecodeError as error:
        position = f"line {error.lineno}, column {error.colno}"
        message = f"The request isn't valid JSON: {error.msg} at {position}."
        raise ValueError(message) from error
    # ...

    try:
        request = DecisionsRequest.model_validate(data)
    except ValidationError as error:
        raise ValueError(_describe(error)) from error
    # ...
    return request


def _reject_repeated_keys(pairs: list[tuple[str, object]]) -> dict[str, object]:
    # json.loads keeps only the last value of a repeated key, so a repeated
    # question ID would drop a question without a word.
    data: dict[str, object] = {}
    for key, value in pairs:
        if key in data:
            raise ValueError(f'The request repeats the key "{key}".')
        data[key] = value
    return data
```

**`evaluation.py`**

```python title="evaluation.py"
# Every API key in an organization shares a limit of 60 requests a minute, so this
# leaves room for other work.
REQUESTS_PER_MINUTE = 50
CONCURRENT_RECORDINGS = 4
MIN_PART_MS = 5_000
MAX_PART_MS = 300_000
MAX_RETRIES = 5

# These errors come from the key or the request, which every recording shares, so
# every other recording would fail the same way.
_BATCH_STOPPING_STATUSES = frozenset({401, 402, 403})
_BATCH_STOPPING_CODES = frozenset({"message_invalid", "question_invalid"})
# These errors come from the file itself, so sending it again unchanged would fail
# the same way.
_FILE_ERROR_STATUSES = frozenset({413, 415})
_FILE_ERROR_CODES = frozenset(
    {
        "audio_decode_failed",
        "audio_format_unsupported",
        "audio_too_short",
        "audio_too_long",
    }
)

# ...


def make_client(
    *,
    requests_per_minute: float = REQUESTS_PER_MINUTE,
    transport: httpx.AsyncBaseTransport | None = None,
) -> AsyncDecisionsClient:
    pacer = _Pacer(interval=60 / requests_per_minute)
    http_client = DefaultAsyncHttpxClient(
        event_hooks={"request": [pacer]}, transport=transport
    )
    return AsyncDecisionsClient(httpx_client=http_client, max_retries=MAX_RETRIES)

# ...


class _Pacer:
    def __init__(self, interval: float) -> None:
        self._interval = interval
        self._next_start = time.monotonic()

    async def __call__(self, _request: httpx.Request) -> None:
        now = time.monotonic()
        start = max(now, self._next_start)
        # Each request claims its start before sleeping, so requests that wait at
        # the same time queue up without a lock.
        self._next_start = start + self._interval
        await asyncio.sleep(start - now)

# ...


async def _run_workers(
    work: Callable[[Stretch, Callable[[_T], None]], Awaitable[None]],
    stretches: Iterable[Stretch],
    stop: asyncio.Event,
) -> AsyncIterator[_T]:
    updates: asyncio.Queue[_T | None] = asyncio.Queue()
    # The workers share one iterator, so each stretch goes to only one of them.
    remaining = iter(stretches)

    async def worker() -> None:
        for stretch in remaining:
            await work(stretch, updates.put_nowait)

    # ...


async def _send_stretch(
    client: AsyncDecisionsClient,
    request: DecisionsRequest,
    stretch: Stretch,
    report: Callable[[Progress], None],
) -> None:
    report(Started(stretch))
    file = await asyncio.to_thread(stretch.path.open, "rb")
    with file:
        # The SDK leaves client.decisions untyped, so this annotation is what lets
        # mypy check the parts.
        parts: AsyncIterator[ChunkResult] = client.decisions.iter_create_chunked(
            request=request,
            file=(stretch.path.name, file),
            max_chunk_ms=MAX_PART_MS,
            start_ms=stretch.start_ms,
            end_ms=stretch.end_ms,
        )
        try:
            async for part in parts:
                report(PartResult(stretch, part))
                if _is_api_error(
                    part.error, _BATCH_STOPPING_STATUSES, _BATCH_STOPPING_CODES
                ):
                    report(BatchStopped(part.error))
                    raise _HaltError
        # The SDK raises ValueError only before it sends anything, when it can't
        # split the file.
        except ValueError as error:
            report(Unreadable(stretch, error))
        else:
            report(Finished(stretch))


async def _estimate_stretch(
    client: AsyncDecisionsClient,
    request: DecisionsRequest,
    stretch: Stretch,
    report: Callable[[EstimateProgress], None],
) -> None:
    try:
        durations = await asyncio.to_thread(_durations, stretch)
    except ValueError as error:
        report(Unreadable(stretch, error))
        return
    for duration_ms in durations:
        # The API rejects a part this short, so Run's request for it would fail
        # without being billed.
        if duration_ms < MIN_PART_MS:
            report(TooShort(stretch, duration_ms))
            continue
        try:
            # The SDK leaves client.decisions untyped, so this annotation is what
            # lets mypy check the response.
            response: InputTokensResponse = await client.decisions.input_tokens(
                audio_duration_ms=duration_ms,
                questions=request.questions,
                context=request.context,
            )
        # The SDK has already retried what can pass, and a total that leaves out
        # parts would understate the run, so any error stops the estimate.
        except (ApiError, httpx.HTTPError) as error:
            report(BatchStopped(error))
            raise _HaltError from error
        report(Estimated(stretch, duration_ms, response.estimated_input_tokens))
```

**`results.py`**

```python title="results.py"
def _same(request: DecisionsRequest, other: DecisionsRequest | None) -> bool:
    # DecisionsRequest equality ignores the order of the questions, but the model
    # answers them together, so a reordered request can answer differently.
    return (
        other is not None
        and request == other
        and list(request.questions) == list(other.questions)
    )


def _resend_from(recording: _Recording) -> int | None:
    if recording.unreadable is not None:
        return None
    start_ms = 0
    for part in recording.parts:
        if _will_resend(part):
            return start_ms
        if part.status == "not_attempted":
            return None
        # The next part starts where this one ends, not at its own start_ms, because
        # an MP3 part's start_ms includes frames repeated from before it.
        start_ms = part.end_ms
    return None if recording.finished else start_ms


def _will_resend(part: ChunkResult) -> bool:
    return part.error is not None and not is_file_error(part.error)

# ...


def _cell(answer: Answer, start_ms: int) -> str:
    match answer:
        case Answer_Noul():
            return f"{answer.noul:.2f}"
        case Answer_Choice():
            return f"{answer.choice} {answer.probabilities[answer.choice]:.2f}"
        case Answer_Score():
            return f"{answer.score:.2f} of {len(answer.legend) - 1}"
        case Answer_Span():
            times = (_clock(start_ms + m.audio_start_ms) for m in answer.spans)
            return ", ".join(times) or "none"
        case _:
            assert_never(answer)
```

## Adapt it to your recordings

1. **Change the default request.** It is `DEFAULT_REQUEST` in `questions.py`. To try a request before you change the code, edit it in the app's **Request** editor. See [Writing questions](/audio-decisions/docs/writing-questions).
2. **Change the pace.** `REQUESTS_PER_MINUTE` in `evaluation.py` sets the pace, `CONCURRENT_RECORDINGS` the number of recordings sent at once and `MAX_PART_MS` the longest part, which must be 5 minutes or less. Sending more recordings at once never raises the pace above `REQUESTS_PER_MINUTE`, and it holds more audio in memory. Every API key in your organization shares a limit of 60 requests a minute. See [Limits](/audio-decisions/docs/limits#rate-limits).
3. **Change how answers are shown.** `_cell` in `results.py` formats each answer's cell. For example, a `score` cell can show the level with the largest share of raters instead of the mean. See [Interpreting answers](/audio-decisions/docs/interpreting-answers).
4. **Keep the results.** `write_downloads` in `results.py` writes the CSV and JSON downloads. The JSON holds the request and, for every part that was sent, its response with its `run_id` or its error. To keep results past the browser tab, save them to your own storage there. See [Store the results](/audio-decisions/docs/evaluating-many-files#store-the-results).
5. **Before you share it.** The app listens only on your own machine and has no sign-in. Anyone who can reach it can spend your API key, so add authentication before you expose it on a network or through a Gradio share link.

## Next steps

#### [Evaluating many files](/audio-decisions/docs/evaluating-many-files)

Prepare a set of recordings and send one request per file within the rate limit.

#### [Interpreting answers](/audio-decisions/docs/interpreting-answers)

Learn how to read each answer and set thresholds for acting on it.