> This page is for Audio Decisions.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://dev.hume.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://dev.hume.ai/_mcp/server.

# Pipecat

**Add Hume Audio Decisions to a Pipecat pipeline with speech-to-text, an LLM, and text-to-speech, so your LLM can reply to how the user sounded, not only to what they said.** `PerceptionReader` sends audio through `HumeAudioClassifier` when voice activity detection (VAD) reports that the user stopped speaking, then attaches the answers to the matching transcription. Each final transcription waits for the answers about the same speech ([inline](/audio-decisions/docs/realtime#inline)) before it reaches the LLM. Clips are cut at the [end of speech](/audio-decisions/docs/realtime#send-at-the-end-of-speech). Pipecat calls the caller "the user".

## Prerequisites

1. Python 3.11 or later.
2. An API key with Hume Audio Decisions access, from the [API keys page](https://cloud.hume.ai/settings/organization/api-keys) of the Hume Platform. See [Authentication](/audio-decisions/docs/authentication) to check or request access.
3. A Pipecat pipeline with speech-to-text, an LLM, and text-to-speech.

## Install

<table>
  <tbody>
    <tr>
      <th width="20%">
        Package
      </th>

      <td>
        [`pipecat-ai[hume]`](https://pypi.org/project/pipecat-ai/)

         on PyPI, version 1.16.0 or later
      </td>
    </tr>

    <tr>
      <th>
        Classifier
      </th>

      <td>
        `pipecat.classifiers.hume.audio_decisions.classifier.HumeAudioClassifier`
      </td>
    </tr>
  </tbody>
</table>

#### pip

```bash
pip install --upgrade "pipecat-ai[hume]>=1.16.0"
```

Set your API key in the environment the bot runs in.

#### Shell

```bash
export HUME_API_KEY=your_api_key
```

* The `hume` extra installs `httpx` with HTTP/2, which `HumeAudioClassifier` uses to call Hume Audio Decisions directly, and Hume's general Python SDK.
* The bot does not need the Hume Audio Decisions SDK from the quickstarts.

## Add it to a pipeline

Three classes connect Hume Audio Decisions to the pipeline:

1. **`PerceptionQuestion`** holds one question, of a type listed in [Map Pipecat questions to primitives](#map-pipecat-questions-to-primitives), and a `description` that tells the LLM how to read its answer.
2. **`PerceptionSource`** holds a classifier and its questions, keyed by name. Pipecat calls each named answer a signal, and its name appears beside its value in the LLM's message.
3. **`PerceptionReader`** asks every source each time VAD reports the end of a speech segment, so several requests can go out before turn detection ends the user's turn. Place it after speech-to-text and before the user aggregator.

Configure VAD on the user aggregator. The reader cuts a clip each time VAD reports the end of speech.

#### bot.py

```python
# Inside your bot's async entry point. transport, stt, llm and tts
# are the services your bot already uses.
import os

from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.classifiers.base_classifier import (
    ChoiceQuestion,
    YesNoQuestion,
)
from pipecat.classifiers.hume.audio_decisions.classifier import (
    HumeAudioClassifier,
)
from pipecat.expressive.processors.perception_reader import (
    PerceptionQuestion,
    PerceptionReader,
    PerceptionSource,
)
from pipecat.pipeline.pipeline import Pipeline
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

hume = PerceptionSource(
    classifier=HumeAudioClassifier(api_key=os.environ["HUME_API_KEY"]),
    questions={
        "tone": PerceptionQuestion(
            question=ChoiceQuestion(
                instructions="How does the speaker sound?",
                options={
                    "neutral": None,
                    "happy": None,
                    "frustrated": None,
                    "angry": None,
                    "sad": None,
                    "confused": None,
                },
            ),
            description="How the user sounded, heard from their voice.",
        ),
        "hesitation": PerceptionQuestion(
            question=YesNoQuestion(
                instructions="Does the speaker sound hesitant or unsure?"
            ),
            description=(
                "Estimated share of raters who would say the user"
                " sounded unsure, from 0 to 1."
            ),
        ),
    },
    timeout=2.0,
)
reader = PerceptionReader(sources=[hume])

context = LLMContext()
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer()
    ),
)

pipeline = Pipeline(
    [
        transport.input(),
        stt,
        reader,
        user_aggregator,
        llm,
        tts,
        transport.output(),
        assistant_aggregator,
    ]
)
```

* **`timeout` on `PerceptionSource`** is the [wait limit](/audio-decisions/docs/realtime#set-the-wait-limit-in-code): how long a transcription waits for the answers, in seconds, counted from when the user stops. The default for an audio classifier is 2 seconds. VAD reports the end after its `stop_secs` of silence, which uses part of that time. Hume Audio Decisions answers in 0.22 seconds, so 2 seconds leaves room for network delay. At the deadline, the reader forwards the transcription with any answers already available. Turn detection, the LLM, and text-to-speech still decide when the bot speaks.
* **`timeout` on `HumeAudioClassifier`** is separate. This `timeout` sets the HTTP client's network timeouts, 10 seconds by default, and retries can extend a request past that timeout. The transcription still continues at the `PerceptionSource` timeout.

## Turn on expressive conversation

> **Info**
>
> Expressive conversation is an alpha feature in Pipecat, and its class names can change between releases.

The LLM reads the answers only with expressive conversation on. Turn it on in the worker's `PipelineParams`. To also control how replies are spoken with expression tags, use a TTS that supports them; the full example uses ElevenLabs Text to Dialogue with `eleven_v4_turbo`. Say what to do with an answer in the expression policy's `guidance`, with thresholds you set on your own calls (see [Interpreting answers](/audio-decisions/docs/interpreting-answers)).

#### bot.py

```python
from pipecat.expressive.policy import ExpressionPolicy
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
from pipecat.workers.runner import WorkerRunner

# Still inside your bot's async entry point, after the pipeline.
worker = PipelineWorker(
    pipeline,
    params=PipelineParams(
        enable_expressive_conversation=True,
        expression_policy=ExpressionPolicy(
            guidance=[
                "If hesitation is 0.7 or above, check that the"
                " user is following before you go on.",
                "If tone is frustrated or angry above 0.6,"
                " acknowledge it before answering.",
            ],
        ),
    ),
)

runner = WorkerRunner()
await runner.add_workers(worker)
await runner.run()
```

#### [Full Pipecat example](https://github.com/pipecat-ai/pipecat/blob/v1.16.0/examples/expressive-conversation/expressive-conversation-hume-audio-decisions.py)

Run the complete bot with turn strategies, a full expression policy, and a choice of transports.

## Check that answers reach the LLM

1. Add the `PerceptionLogger` from [Use answers without waiting for the turn](#use-answers-without-waiting-for-the-turn).
2. Speak for at least 5 seconds, then pause long enough for VAD to report the end of speech, and confirm that the logger prints the answers.
3. Inspect the user message sent to the LLM. The message starts with the answers, as in `([hesitation value=0.74]) I'm fine.`
4. If the logger prints answers but the message has none, check `enable_expressive_conversation`.
5. If the logger prints nothing, look for error frames and for `HTTP 401` or `403` in the log, which mean the key is rejected or lacks access. Clips shorter than 1 second are not sent.

## How it sends audio

`PerceptionReader` follows the [live audio guidance](/audio-decisions/docs/preparing-your-audio#live-audio):

<table>
  <thead>
    <tr>
      <th>
        Choice in Realtime decisions
      </th>

      <th>
        What the reader does
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        [Which audio to send](/audio-decisions/docs/realtime#choose-which-audio-to-send)
      </td>

      <td>
        Speech segments of the user, as 16 kHz mono WAV, extended to 5 seconds within the turn.
      </td>
    </tr>

    <tr>
      <td>
        [When to send](/audio-decisions/docs/realtime#choose-when-to-send)
      </td>

      <td>
        At each end of speech that VAD reports, before turn detection decides the turn is over.
      </td>
    </tr>

    <tr>
      <td>
        [Whether the reply waits](/audio-decisions/docs/realtime#choose-whether-the-reply-waits)
      </td>

      <td>
        Inline. The transcription waits up to 

        `timeout`

        , 2 seconds by default. When it is released, it includes the answers already available; answers still pending are left out of that turn.
      </td>
    </tr>

    <tr>
      <td>
        Requests at a time
      </td>

      <td>
        One. Of the clips that arrive while a request runs, only the newest is sent next.
      </td>
    </tr>

    <tr>
      <td>
        Other uses of an answer
      </td>

      <td>
        Every answer is also broadcast as a 

        `PerceptionFrame`

        , including late ones. See 

        [Use answers without waiting for the turn](#use-answers-without-waiting-for-the-turn)

        .
      </td>
    </tr>
  </tbody>
</table>

Clips the reader cuts at each end of speech

User Speech

Bot

Bot speaks

Clip 1 4.2 s

Clip 2 5 s

Clip 3 2.7 s

0 s 5 10 15 s

Example times. Clip 3 follows the bot's turn, so it is sent on its own.

* Each time VAD reports the end of speech, the reader cuts a clip of the speech that just ended. A clip starts up to 1 second before the speech and ends with the short silence VAD waits for before reporting the end.
* A clip shorter than 5 seconds is extended toward 5 seconds with the end of the user's earlier speech, as far back as the bot's last turn. Silence is never added to reach 5 seconds. A clip still shorter than 1 second after that is not sent.
* When the bot stops speaking, the reader drops the user's earlier speech, so later clips are not extended with it. If the user is still speaking when the bot stops, that speech is kept. Muting also drops the earlier speech, and audio received while muted is not kept.
* No clip is longer than 5 minutes.
* When the bot has already spoken, including a greeting before the user's first turn, the reader sends the bot's last reply as the request's `context`, such as `{"assistant": "Let me check that booking."}`. The classifier serializes that context as JSON and keeps its last 1,000 characters, so a long reply loses the start of the JSON text. Otherwise no `context` is sent. The reader always sets `context` itself, so you cannot add your own description of the call.

## How the answers reach the LLM

* The answers go at the head of the user's message, as in `([tone sad=0.81, neutral=0.12],[hesitation value=0.74]) I'm fine.` A `ChoiceQuestion` reports its likeliest options, two by default (set `max_labels` on the `PerceptionQuestion`). A `YesNoQuestion` reports its value from 0 to 1 and a `ScoreQuestion` its score, both written as `value=`.
* When several clips are answered within one user turn, the user message combines them: each value keeps its highest reading, and each label keeps its highest reading across clips. A combined message can list more than `max_labels` labels, and its values summarize several clips rather than one answer about the whole turn.

## Handle setup and request failures

* If the connection check at start-up fails, `HumeAudioClassifier` logs a warning, and requests are still attempted. A temporary connection problem can recover on a later request.
* A failed request emits an error frame, and the transcription goes on without that source's answers. A rejected key fails every request until you correct it.
* If a processor becomes unusable, the bot carries on without it by default. To end the call instead, import `ProcessorUnusablePolicy` from `pipecat.pipeline.worker` and pass `processor_unusable_policy=ProcessorUnusablePolicy.END` to `PipelineWorker`.

## Use answers without waiting for the turn

Every answer is also broadcast in a `PerceptionFrame` as soon as it arrives, including answers that arrive after the timeout. Any processor in the pipeline receives the frame; place yours right after the reader to act on answers in your backend, as on the [background](/audio-decisions/docs/realtime#background) path. Listening for these frames does not change how long the transcription waits, which `PerceptionSource.timeout` still sets.

#### bot.py

```python
from loguru import logger

from pipecat.expressive.perception import format_perception
from pipecat.frames.frames import Frame, PerceptionFrame
from pipecat.processors.frame_processor import (
    FrameDirection,
    FrameProcessor,
)


class PerceptionLogger(FrameProcessor):
    def __init__(self):
        super().__init__()
        self._streak = 0

    async def process_frame(
        self, frame: Frame, direction: FrameDirection
    ):
        await super().process_frame(frame, direction)
        if isinstance(frame, PerceptionFrame):
            logger.info(format_perception(frame.perception))
            for signal in frame.perception:
                if signal.name != "hesitation":
                    continue
                if (signal.value or 0) >= 0.7:
                    self._streak += 1
                else:
                    self._streak = 0
                if self._streak == 2:  # Once per run of answers.
                    self.create_task(notify_supervisor())
        await self.push_frame(frame, direction)
```

1. Put an instance right after the reader: `reader, PerceptionLogger(), user_aggregator`.
2. Push every frame on, as the example does, so the pipeline keeps moving.
3. Act on a run of answers rather than a single clip, as in [Act on answers](/audio-decisions/docs/realtime#act-on-answers). The example notifies once when two clips in a row pass.
4. Start slow work such as `notify_supervisor()`, your own function, with `create_task` instead of awaiting it in `process_frame`.

## Map Pipecat questions to primitives

Each Pipecat question type is sent as one [Hume Audio Decisions primitive](/audio-decisions/docs/primitives). A yes or no value estimates the share of raters who would answer yes; set thresholds for it on your own calls as in [Interpreting answers](/audio-decisions/docs/interpreting-answers#setting-thresholds).

<table>
  <thead>
    <tr>
      <th>
        Pipecat question
      </th>

      <th>
        Hume Audio Decisions primitive
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        `YesNoQuestion`

        , with optional 

        `yes`

         and 

        `no`

         criteria
      </td>

      <td>
        [`noul`](/audio-decisions/docs/primitives#noul)
      </td>
    </tr>

    <tr>
      <td>
        `ChoiceQuestion`

        , with an optional description per option
      </td>

      <td>
        [`choice`](/audio-decisions/docs/primitives#choice)
      </td>
    </tr>

    <tr>
      <td>
        `ScoreQuestion`
      </td>

      <td>
        [`score`](/audio-decisions/docs/primitives#score)
      </td>
    </tr>
  </tbody>
</table>

## Next steps

#### [Realtime decisions](/audio-decisions/docs/realtime)

Compare inline and background answers and the ways to cut a caller's audio.

#### [Writing questions](/audio-decisions/docs/writing-questions)

Phrase each question as one judgment a listener would make on hearing the turn.

#### [Interpreting answers](/audio-decisions/docs/interpreting-answers)

Set the thresholds in your expression policy on your own calls.