Pipecat
Add Hume Audio Decisions to a Pipecat pipeline with speech-to-text, an LLM, and text-to-speech, so your LLM can reply to how the user sounded, not only to what they said. PerceptionReader sends audio through HumeAudioClassifier when voice activity detection (VAD) reports that the user stopped speaking, then attaches the answers to the matching transcription. Each final transcription waits for the answers about the same speech (inline) before it reaches the LLM. Clips are cut at the end of speech. Pipecat calls the caller “the user”.
Prerequisites
- Python 3.11 or later.
- An API key with Hume Audio Decisions access, from the API keys page of the Hume Platform. See Authentication to check or request access.
- A Pipecat pipeline with speech-to-text, an LLM, and text-to-speech.
Install
Set your API key in the environment the bot runs in.
- The
humeextra installshttpxwith HTTP/2, whichHumeAudioClassifieruses to call Hume Audio Decisions directly, and Hume’s general Python SDK. - The bot does not need the Hume Audio Decisions SDK from the quickstarts.
Add it to a pipeline
Three classes connect Hume Audio Decisions to the pipeline:
PerceptionQuestionholds one question, of a type listed in Map Pipecat questions to primitives, and adescriptionthat tells the LLM how to read its answer.PerceptionSourceholds a classifier and its questions, keyed by name. Pipecat calls each named answer a signal, and its name appears beside its value in the LLM’s message.PerceptionReaderasks every source each time VAD reports the end of a speech segment, so several requests can go out before turn detection ends the user’s turn. Place it after speech-to-text and before the user aggregator.
Configure VAD on the user aggregator. The reader cuts a clip each time VAD reports the end of speech.
timeoutonPerceptionSourceis the wait limit: how long a transcription waits for the answers, in seconds, counted from when the user stops. The default for an audio classifier is 2 seconds. VAD reports the end after itsstop_secsof silence, which uses part of that time. Hume Audio Decisions answers in 0.22 seconds, so 2 seconds leaves room for network delay. At the deadline, the reader forwards the transcription with any answers already available. Turn detection, the LLM, and text-to-speech still decide when the bot speaks.timeoutonHumeAudioClassifieris separate. Thistimeoutsets the HTTP client’s network timeouts, 10 seconds by default, and retries can extend a request past that timeout. The transcription still continues at thePerceptionSourcetimeout.
Turn on expressive conversation
Expressive conversation is an alpha feature in Pipecat, and its class names can change between releases.
The LLM reads the answers only with expressive conversation on. Turn it on in the worker’s PipelineParams. To also control how replies are spoken with expression tags, use a TTS that supports them; the full example uses ElevenLabs Text to Dialogue with eleven_v4_turbo. Say what to do with an answer in the expression policy’s guidance, with thresholds you set on your own calls (see Interpreting answers).
Check that answers reach the LLM
- Add the
PerceptionLoggerfrom Use answers without waiting for the turn. - Speak for at least 5 seconds, then pause long enough for VAD to report the end of speech, and confirm that the logger prints the answers.
- Inspect the user message sent to the LLM. The message starts with the answers, as in
([hesitation value=0.74]) I'm fine. - If the logger prints answers but the message has none, check
enable_expressive_conversation. - If the logger prints nothing, look for error frames and for
HTTP 401or403in the log, which mean the key is rejected or lacks access. Clips shorter than 1 second are not sent.
How it sends audio
PerceptionReader follows the live audio guidance:
Clips the reader cuts at each end of speech
- Each time VAD reports the end of speech, the reader cuts a clip of the speech that just ended. A clip starts up to 1 second before the speech and ends with the short silence VAD waits for before reporting the end.
- A clip shorter than 5 seconds is extended toward 5 seconds with the end of the user’s earlier speech, as far back as the bot’s last turn. Silence is never added to reach 5 seconds. A clip still shorter than 1 second after that is not sent.
- When the bot stops speaking, the reader drops the user’s earlier speech, so later clips are not extended with it. If the user is still speaking when the bot stops, that speech is kept. Muting also drops the earlier speech, and audio received while muted is not kept.
- No clip is longer than 5 minutes.
- When the bot has already spoken, including a greeting before the user’s first turn, the reader sends the bot’s last reply as the request’s
context, such as{"assistant": "Let me check that booking."}. The classifier serializes that context as JSON and keeps its last 1,000 characters, so a long reply loses the start of the JSON text. Otherwise nocontextis sent. The reader always setscontextitself, so you cannot add your own description of the call.
How the answers reach the LLM
- The answers go at the head of the user’s message, as in
([tone sad=0.81, neutral=0.12],[hesitation value=0.74]) I'm fine.AChoiceQuestionreports its likeliest options, two by default (setmax_labelson thePerceptionQuestion). AYesNoQuestionreports its value from 0 to 1 and aScoreQuestionits score, both written asvalue=. - When several clips are answered within one user turn, the user message combines them: each value keeps its highest reading, and each label keeps its highest reading across clips. A combined message can list more than
max_labelslabels, and its values summarize several clips rather than one answer about the whole turn.
Handle setup and request failures
- If the connection check at start-up fails,
HumeAudioClassifierlogs a warning, and requests are still attempted. A temporary connection problem can recover on a later request. - A failed request emits an error frame, and the transcription goes on without that source’s answers. A rejected key fails every request until you correct it.
- If a processor becomes unusable, the bot carries on without it by default. To end the call instead, import
ProcessorUnusablePolicyfrompipecat.pipeline.workerand passprocessor_unusable_policy=ProcessorUnusablePolicy.ENDtoPipelineWorker.
Use answers without waiting for the turn
Every answer is also broadcast in a PerceptionFrame as soon as it arrives, including answers that arrive after the timeout. Any processor in the pipeline receives the frame; place yours right after the reader to act on answers in your backend, as on the background path. Listening for these frames does not change how long the transcription waits, which PerceptionSource.timeout still sets.
- Put an instance right after the reader:
reader, PerceptionLogger(), user_aggregator. - Push every frame on, as the example does, so the pipeline keeps moving.
- Act on a run of answers rather than a single clip, as in Act on answers. The example notifies once when two clips in a row pass.
- Start slow work such as
notify_supervisor(), your own function, withcreate_taskinstead of awaiting it inprocess_frame.
Map Pipecat questions to primitives
Each Pipecat question type is sent as one Hume Audio Decisions primitive. A yes or no value estimates the share of raters who would answer yes; set thresholds for it on your own calls as in Interpreting answers.