Primitives

A primitive is the type of a question, and it sets what the answer describes. Answers to noul, choice and score questions estimate how a panel of human raters would respond to the question: each probability is the estimated share of raters who would give that answer. A span answer locates an event in the audio and gives the time spans where it may occur.

What the numbers describe

A probability estimates the share of human raters who would give that answer, so a value near 0.5 means raters would split, not that the model is unsure. confidence measures how much raters would agree, not how certain the model is.

What the model reads

For each question, the model reads:

1. The question’s instructions.
2. The question’s criteria: a noul question’s true and false guidance, each choice option’s name with its description when it has one, or the text of each score level.
3. The request’s context.
4. The audio.

A noul, choice or score answer estimates how raters would respond if they heard the audio and read the same question, criteria and context. All the questions in a request are answered in one model inference, so an answer can change when other questions are added, removed or reordered. To compare answers across recordings, send the same questions in the same order.

Each question is keyed by an ID you choose, such as agent_tone, which can be any non-empty string of up to 256 characters. The response returns each answer under the same ID.

Choosing a primitive

1. Use noul for a yes or no question. A noul answer is a share of raters, not a degree: 0.6 means an estimated 60% of raters would answer yes, not that the answer is “somewhat yes”. To measure degree, such as how natural a voice sounds, use score.
2. Use choice when the options have no order, and score when they do. Tone (warm, neutral, curt) is a choice. Naturalness from “Not at all” to “Extremely” is a score, because its levels have an order that the answer’s mean level and confidence take into account.
3. Use span to find when an event occurs, and noul to ask whether it occurs. A span answer gives times but no share of raters, and a noul answer gives a share of raters but no times.

noul

A noul question asks a yes or no question. Each rater would answer yes or no, and the answer is the estimated share who would answer yes. The name comes from the Bernoulli distribution, which describes a single yes or no outcome.

criteria is optional. Add it when what counts as yes or no is not clear from the question alone.

{
  "questions": {
    "talks_over_caller": {
      "type": "noul",
      "instructions": "Does the agent talk over the caller?",
      "criteria": {
        "true": "The agent starts speaking while the caller is still mid-sentence.",
        "false": "The agent waits for the caller to finish, or overlaps only with a brief acknowledgement such as \"mm-hmm\"."
      }
    }
  }
}

{
  "talks_over_caller": {
    "type": "noul",
    "noul": 0.1274
  }
}

An estimated 13% of raters would say the agent talks over the caller. A noul answer has no confidence, because the share already shows how raters would split: they agree most when noul is near 0 or 1, and least when it is near 0.5.

choice

A choice question asks which of two or more unordered options applies. Each rater would choose one option, and the answer gives the estimated share of raters who would choose each. The options are presented to the model in the order listed.

{
  "questions": {
    "agent_tone": {
      "type": "choice",
      "instructions": "Which best describes the tone of the agent?",
      "criteria": [
        {
          "name": "warm",
          "description": "Warm and friendly toward the caller"
        },
        {
          "name": "neutral",
          "description": "Polite but impersonal"
        },
        {
          "name": "curt",
          "description": "Short or dismissive with the caller"
        }
      ]
    }
  }
}

{
  "agent_tone": {
    "type": "choice",
    "choice": "warm",
    "confidence": 0.3072,
    "probabilities": {
      "warm": 0.5381,
      "neutral": 0.3926,
      "curt": 0.0693
    }
  }
}

About 54% of raters would call the agent warm and 39% neutral. warm is the most common choice, but raters would split between the first two options.

confidence measures how far the top option’s share stands above an even split across all options. It is 0 when every option is equally likely, and 1 when every rater would choose the same option. With n options and a top probability of p_max:

confidence = (p_max - 1/n) / (1 - 1/n)

For the answer above, (0.5381 - 1/3) / (1 - 1/3) gives 0.3072. Only the top share counts, so the split among the other options does not change confidence.

score

A score question asks where the audio falls on a scale of two or more ordered levels. Each rater would choose one level, and the answer gives the estimated share of raters who would choose each.

List the levels in criteria from lowest to highest. They are indexed from 0, and the model reads each level’s text.

{
  "questions": {
    "naturalness": {
      "type": "score",
      "instructions": "How natural does the voice sound?",
      "criteria": ["Not at all", "2", "3", "4", "Extremely"]
    }
  }
}

{
  "naturalness": {
    "type": "score",
    "score": 2.9167,
    "confidence": 0.4503,
    "probabilities": {
      "0": 0.0186,
      "1": 0.0642,
      "2": 0.1873,
      "3": 0.4417,
      "4": 0.2882
    },
    "legend": {
      "0": "Not at all",
      "1": "2",
      "2": "3",
      "3": "4",
      "4": "Extremely"
    }
  }
}

Level indices count from 0, so they differ from this scale’s labels: index 1 is the level labeled “2”, and index 3 is the level labeled “4”. legend maps each index back to its text.

The most common rating is index 3 (“4”), chosen by about 44% of raters. The mean level of 2.9167 sits just below index 3, pulled down by the raters who would choose lower levels.

The same score can come from different splits. A score of 2 on this scale results when most raters would choose index 2, and also when raters would split between the two ends of the scale. Read probabilities and confidence to tell these apart.

confidence measures how closely raters would cluster around the median level, compared with raters spread evenly across every level. Shares one level away from the median lower it less than shares at the far end of the scale. It is 1 when every rater would choose the same level, and 0 when the shares are at least as spread out as an even split. With p_i the share for level i, m the median level, and MAD_u the mean absolute deviation of an even split over the same levels:

confidence = max(0, 1 - sum_i(p_i * |i - m|) / MAD_u)

For the answer to “How natural does the voice sound?”, the median level is 3, the weighted distance from it is 0.6597, and MAD_u for five levels is 1.2, which gives 1 - 0.6597 / 1.2 = 0.4503. In the two answers with a score of 2, confidence is 0.68 when most raters would choose index 2, and 0 when raters would split between the two ends.

span

A span question asks when an event occurs in the audio. It takes no criteria. The answer lists the time spans where the model locates the event, each with the model’s estimate that the event occurs in it. span is experimental: ongoing training is refining how precisely the model locates events, and the question and answer formats are stable. It is most reliable for vocal bursts, such as laughs and sighs, and for sound events, such as a phone ringing.

{
  "questions": {
    "vocal_bursts": {
      "type": "span",
      "instructions": "When does the speaker laugh, sigh, gasp or make another vocal burst?"
    }
  }
}

{
  "vocal_bursts": {
    "type": "span",
    "spans": [
      {
        "audio_start_ms": 9840,
        "audio_end_ms": 10630,
        "probability": 0.9127
      },
      {
        "audio_start_ms": 27450,
        "audio_end_ms": 28390,
        "probability": 0.8364
      }
    ]
  }
}

The model locates vocal bursts at about 9.8 and 27.5 seconds into the file.

Invalid questions

A question that does not match its primitive’s schema is rejected with question_invalid, and param points to where the error was found, which can be the question rather than the field. This includes a choice question with fewer than two options or more than 20, or with duplicate names, a score question with fewer than two levels or more than 10, or with duplicate levels, and a question whose instructions or criteria text is longer than its limit. See Limits and Errors.

Next steps