Preparing your audio

Each request is answered on its own, with no memory of earlier requests, so everything an answer depends on has to be in the audio you send with it. The same question can mean different things on different audio: “Does the caller sound frustrated?” asks something different of one sentence, one turn and a whole call.

Listen to a few of the files you plan to send, after you prepare them. If a person would find it hard to hear or judge what a question asks about, such as a quiet voice under loud noise, the model is likely to find it hard too. Listening also catches problems that preparing the audio can add, such as clipped words or abrupt joins.

Choose whose voice

First decide whether an answer can take everyone in the recording into account, or must judge one person on their own. Most questions take the first path, which needs no editing.

Ask about the whole conversation

Use this path for questions about the conversation itself, such as who interrupts, for questions about one person where the other voices can count, and for recordings of one speaker.

1. Send the recording as it is, as one file. A stereo file is mixed into one channel. See Convert the file.
2. Name the person each question is about, such as “the caller”, and describe each person in context.
3. Add a transcript if you have one. It is optional. Put it in context on its own line, labeled Transcript:.

Ask about one person on their own

Use this path when an answer must not be affected by anyone else’s voice, such as how frustrated the caller sounds when the agent is loud. Every cut and join adds sounds that were not in the conversation, and the model hears them like any other audio, so listen to a few of the files you prepare.

1. Find the person’s audio.
   • In a stereo recording with one person on each channel, take that person’s channel, as in Convert the file. Check which channel is which on a few files before sending the rest, since the order varies between telephony providers.
   • In a recording with everyone on one channel, find the person’s turns with speaker diarization, or with the timestamps of a transcript that labels each speaker.
2. On one channel, cut out each of their turns.
   • Cut in the silence before and after the speech, not at the first and last word, so no word is clipped.
   • Fade in and out over a few milliseconds at each cut, so the cuts do not click.
   • Leave out stretches where the other person talks at the same time. They still hold the other voice, and diarization is least reliable there.
3. Choose how to send them.
   • Send them as one file for one answer that describes the person across all their speech. The gaps where others spoke can stay as silence, or you can shorten them to make the file shorter, which costs less and fits more speech into 5 minutes. Keep a shortened gap about as long as a pause between the person’s own sentences: shorter gaps make them sound abrupt or rushed, which changes answers about pace, tone or hesitation.
   • Send each turn of 5 seconds or more as its own request for an answer about each turn.

On either path

1. The model does not hear which side or channel a sound comes from, so do not ask about left or right.
2. Overlapping speech is hard to judge either way. To find it, ask about it directly on a short stretch of audio.

Choose how much time

Whether something happens in a turn or a call

Send the whole file, or the turn when it is at least 5 seconds long. You need one answer, and cutting the audio into windows multiplies requests. A turn shorter than 5 seconds is rejected with audio_too_short, so extend it with the audio before and after it.

How a known moment sounds

1. When you know where the moment is, such as the caller’s reply to a yes or no question or the agent’s greeting, cut a window centered on it, at least 5 seconds long, since shorter audio is rejected with audio_too_short.
2. A one-word reply lasts a fraction of a second, so keep the audio around it.

Where or how often something happens

Overlapping windows with a noul question, for frequent events or when missing one is costly:

1. Make each window at least twice as long as the longest event you expect, and at least 5 seconds: 5 seconds for filler words such as “um”, 6 to 8 seconds for laughter.
2. Overlap the windows so every event fits whole inside at least one. For example, 6-second windows every 3 seconds contain any event up to 3 seconds long. Half-overlapping windows double the number of requests.
3. Cut at pauses where you can, using voice activity detection.
4. Treat yes answers in adjacent windows as one event.

A span question, checked with noul, for rare events in long audio:

1. Ask the span question of the turn or the file.
2. For each span returned, cut a window of at least 5 seconds centered on it and ask a noul question about the same event.
3. Keep the spans the noul answer confirms.

audio_start_ms is measured from the start of the file you sent. When the file is cut from a longer recording, add the cut’s start time. See Primitives for the fields of a span answer.

An overall impression of a long recording

1. A request takes 5 seconds to 5 minutes of audio. Split longer recordings into parts, ask every part the same questions and combine the answers in your code. Evaluating many files shows how to split them.
2. Split at pauses or turn ends rather than every 5 minutes, so no turn is cut in half.
3. Combine with the mean for an overall level, or with the maximum for whether something happens in any part. Weight by length when parts differ a lot.
4. Each part is answered without the others, so a combined answer can differ from how a listener would judge the whole recording.

Live audio

1. Clips: send short clips of the latest audio as the conversation goes on. Each clip is an ordinary request.
2. Window length: at least 5 seconds, and long enough to answer on its own. 5 seconds is a reasonable start for how someone sounds.
3. How often to send: no more often than one request takes to return. If a request is still waiting when the next clip is due, skip that clip.
4. At the end of each turn, or on a timer: sending the latest 5 seconds or more when the speaker pauses often works better, since the clip ends on a whole thought.
5. Rate limit: the organization’s 60 requests per minute cover every stream. One clip every 5 seconds is 12 requests per minute, so five streams at that rate reach the limit. See Limits.
6. Smoothing: act on several clips in a row, not one.
7. Silence and mute: skip clips that are mostly silence, and never join audio from before and after a mute.
8. Calls: send both sides mixed into one clip. Send each side as its own clip only to judge one person on their own.

The Answering live audio cookbook runs an app that records short clips as audio plays and sends each clip as it ends.

Decide what stays in the audio

1. Silence: trim it for questions about the speech itself, such as tone or emotion, keeping at least 5 seconds of audio. Keep it for questions about pace, pauses or response timing.
2. Background sound: keep it for questions about the environment. Do not remove noise before asking about recording quality.
3. Re-encoding: each lossy re-encode can make audio sound more artificial. Send the original where you can, and avoid chains such as MP3 to WAV to MP3.
4. Audio processing: turn off noise suppression, echo cancellation and automatic gain control where you can. They change the loudness and background noise that questions ask about.

Convert the file

1. The API takes MP3 or WAV. See Limits.
2. The model hears audio as 16 kHz, 16-bit, one channel. A file with two channels is mixed into one before the model hears it.
3. The SDKs’ create converts WAV files to 16 kHz mono for you. To judge one person on their own, pass downmix_strategy="first_channel" in Python or downmixStrategy: "first_channel" in TypeScript to keep only the first channel.
4. Without the SDKs, convert large WAV files to 16 kHz mono so they fit the size limit.

# Convert any file to 16 kHz mono WAV, mixing all channels into one
ffmpeg -i in.m4a -ac 1 -ar 16000 -sample_fmt s16 out.wav
# Keep only the left channel, to judge that person on their own
ffmpeg -i call.wav -af "pan=mono|c0=c0" -ar 16000 -sample_fmt s16 left.wav
# Keep only the right channel
ffmpeg -i call.wav -af "pan=mono|c0=c1" -ar 16000 -sample_fmt s16 right.wav
# Cut a 6-second window starting at 12.5 seconds
ffmpeg -ss 12.5 -t 6 -i left.wav -c copy window.wav
# Cut one speaker's turn from 12.1 to 18.1 seconds, with 10 ms fades so the edges do not click
ffmpeg -ss 12.1 -t 6 -i call.wav -af "afade=t=in:d=0.01,afade=t=out:st=5.99:d=0.01" -ac 1 -ar 16000 -sample_fmt s16 turn.wav

Next steps