Writing questions

Ask each question for one focused judgment about the audio, and combine the answers in your code to make larger decisions.

Build a decision from several questions

1. Ask for judgments a listener makes on hearing the audio, and split broad ones into parts. “How frustrated does the caller sound?” is one judgment. “Did the agent handle the call well?” depends on several, such as the agent’s tone and whether the caller asks to cancel, so ask one question for each and combine the answers in your code.
2. Ask every question you might need in one request. The questions in a request are answered in one model inference, so an answer can change when other questions are added, removed or reordered. Send every recording the same questions in the same order, and ignore the answers you do not need.
3. Use a second request when one answer decides the next question. For example, a choice question can find out why someone called, and a second request with the same file can ask questions specific to that reason.

Every question

1. Ask about something a person could answer by listening, such as whether the caller sounds frustrated or whether there is background noise. Listen to a few of your recordings and check that you could answer the question yourself. If a person would find it hard to answer, the model is likely to find it hard too.
2. Ask about one thing. “Does the agent sound warm and professional?” asks about two qualities in one answer. Ask “Does the agent sound warm?” and “Does the agent sound professional?” as separate questions.
3. Say who. In a recording of more than one person, name the person the question is about, such as “the agent” or “the caller”, and describe each person in context. To judge one person apart from the others, send that person’s audio as its own file. See Ask about one person on their own.
4. Ask neutrally. Ask “How patient does the agent sound?” rather than “Isn’t the agent impatient?”, which suggests an answer.
5. Put the text a question depends on in context, such as the script that was read, the style the speech was asked for or a description of the voice. Put each piece on its own line with a label, such as Script:. Every question in a request shares the same context, so send questions that need different context in separate requests.

Each question type

noul

1. Phrase the question so that yes is the case your code acts on. For example, ask “Is there distracting background noise?”, so that a high value means a problem.
2. Add criteria when the line between yes and no is subtle, such as whether an accent counts as a mispronunciation. Compare answers with and without criteria to see whether it helps.

choice

1. Give options that exclude each other and cover every outcome. Add an option such as “none of these” for audio that fits none of the others.
2. Include a neutral option. Add “neutral” to emotions such as angry, fearful and sad, and “unsure” to yes and no, so that audio with no clear answer has an option to go to.
3. Name distinct categories instead of narrow number ranges. Options such as “slow”, “normal” and “fast” work better than “100 to 120 words per minute” and “120 to 140 words per minute”, since a listener can tell the categories apart but not the ranges.
4. Ask a noul question for each quality that can co-occur. The probabilities in a choice answer sum to 1, so options that can both be true, such as “friendly” and “professional”, split the probability between them.
5. Add a description when an option’s name could be read more than one way, such as “neutral” described as “Polite but impersonal”.

score

Use a common five-point scale, with the ends labeled and the levels between them labeled or numbered.

1. Keep each scale to one quality. A scale from “Calm and slow” to “Agitated and fast” combines two, so ask one score question for each.
2. List levels from lowest to highest. The answer’s score is the mean level, so a higher score means more of the quality.

span

1. Name the event in full, whether it is a sound, such as a phone ringing, or something said, such as the caller asking to cancel. A span question has no criteria, so its instructions carry the whole description.
2. Expect the most reliable answers for vocal bursts and sound events, such as a laugh, a sigh or a phone ringing. For other events, such as a word being spoken, check the spans on audio you know before you rely on them.
3. List examples when the event covers several kinds of sound, such as “laugh, sigh, gasp or make another vocal burst”.

Examples

Each example is the JSON sent as request for one use case.

Evaluating text-to-speech output

This request evaluates one output of a text-to-speech model. The context gives the script and the requested style.

{
  "context": "Output from a text-to-speech model under evaluation.\nScript: Thanks for calling. Your order shipped this morning and should arrive on Thursday.\nRequested style: Warm and upbeat.",
  "questions": {
    "naturalness": {
      "type": "score",
      "instructions": "How natural does the voice sound?",
      "criteria": ["Not at all", "2", "3", "4", "Extremely"]
    },
    "follows_style": {
      "type": "score",
      "instructions": "The delivery matches the requested style.",
      "criteria": ["Strongly disagree", "Disagree", "Neutral", "Agree", "Strongly agree"]
    },
    "mispronounced": {
      "type": "noul",
      "instructions": "Does the voice mispronounce a word in the script?",
      "criteria": {
        "true": "At least one word is pronounced wrongly, such as a name or a number read incorrectly.",
        "false": "Every word is pronounced correctly. An accent does not count as a mispronunciation."
      }
    },
    "artifacts": {
      "type": "noul",
      "instructions": "Are there glitches, artifacts or unwanted changes in the voice?"
    }
  }
}

1. follows_style rates a statement on an agreement scale, against the style given in context.
2. mispronounced checks the speech against the script in context, and its criteria say that an accent does not count.

Reviewing support calls

This request reviews a recorded support call. The context says who the agent and the caller are.

{
  "context": "A support call to an internet provider. The agent answers the call, and the caller reports an outage.",
  "questions": {
    "agent_tone": {
      "type": "choice",
      "instructions": "Which best describes the tone of the agent?",
      "criteria": [
        {
          "name": "warm",
          "description": "Warm and friendly toward the caller"
        },
        {
          "name": "neutral",
          "description": "Polite but impersonal"
        },
        {
          "name": "curt",
          "description": "Short or dismissive with the caller"
        }
      ]
    },
    "caller_frustration": {
      "type": "score",
      "instructions": "How frustrated does the caller sound?",
      "criteria": ["Not at all", "2", "3", "4", "Extremely"]
    },
    "background_noise": {
      "type": "noul",
      "instructions": "Is there distracting background noise?"
    },
    "cancel_request": {
      "type": "span",
      "instructions": "When does the caller ask to cancel their service?"
    }
  }
}

caller_frustration asks how much, so it is a score question. Interpreting answers combines this request’s answers into one decision.

Finding events in a recording

This request finds vocal bursts and a ringing phone in a recording.

{
  "questions": {
    "vocal_bursts": {
      "type": "span",
      "instructions": "When does the speaker laugh, sigh, gasp or make another vocal burst?"
    },
    "phone_ringing": {
      "type": "span",
      "instructions": "When does a phone ring?"
    }
  }
}

Questions the model does not answer well

1. Questions that need musical understanding, such as where the chorus starts or when the guitars come in.
2. Open questions, such as “Why is the caller calling?”. Answers are probabilities over the options you give, or time spans, so ask a choice question that lists the reasons callers contact you.
3. Where a sound comes from, such as left or right. The model hears one channel.
4. Sound above 8 kHz. The model hears audio at a 16 kHz sample rate, which keeps frequencies up to 8 kHz.
5. Silent or very short audio. The model judges silence like any other audio and can answer it with high confidence, and a very short clip gives it little to judge. Skip files that are silent or nearly silent.
6. Overall quality of phone audio. Phone audio has less high-frequency sound, so the model hears it as lower in quality and “Does the audio sound muffled?” can come out high for nearly every call. Ask about a specific problem, such as background noise, and compare quality only between recordings of the same kind.

Check your questions on audio you know

Before you rely on a question, run it on audio where you already know the answer.

1. Test on labeled audio of the kind you will judge. Include clear cases at each end, such as an obviously angry call and an obviously calm one, and compare the answers with your labels at the thresholds you plan to use. A question that works on studio recordings can behave differently on phone calls, and synthetic voices can be easier to judge than recordings of people.
2. Ask opposite questions. “How calm does the caller sound?” and “How stressed does the caller sound?” should move in opposite directions across clips. If they rise together, the answers are following something else, such as loudness.
3. Try a second wording. Two wordings of the same question should agree on most clips. If they disagree, make the question more specific.
4. Listen to the clips with the highest and lowest answers, and to a few in between.

Next steps