Empathic Voice Interface (EVI)

Tool use

Guide to using function calling with the Empathic Voice Interface (EVI).

EVI simplifies the integration of external APIs through function calling. Developers can integrate custom functions that are invoked dynamically based on the user’s input, enabling more useful conversations. There are two key concepts for using function calling with EVI, Tools and Configurations (Configs):

  • Tools are resources that EVI uses to do things, like search the web or call external APIs. For example, tools can check the weather, update databases, schedule appointments, or take actions based on what occurs in the conversation. While the tools can be user-defined, Hume also offers natively implemented tools, like web search, which are labeled as “built-in” tools.

  • Configurations enable developers to customize an EVI’s behavior and incorporate these custom tools. Setting up an EVI configuration allows developers to seamlessly integrate their tools into the voice interface. A configuration includes prompts, user-defined tools, and other settings.

Currently, our function calling feature only supports OpenAI models. Function calling is not available if you are using your own custom language model. We plan to support more function calling LLMs in the future.

The focus of this guide is on creating a Tool and a Configuration that allows EVI to use the Tool. Additionally, this guide details the message flow of function calls within a session, and outlines the expected responses when function calls fail. Refer to our Configuration Guide for detailed, step-by-step instructions on how to create and use an EVI Configuration.

Explore this sample project for an example of how Tool use could be implemented in practice.

Setup

For EVI to leverage tools or call functions, a configuration must be created with the tool’s definition. Our step-by-step guide below walks you through creating a tool and a configuration.

Create a Tool

We will first create a Tool with a specified function. In this case, we will create a tool for getting the weather. Create this tool by making a POST request to /tools with the following request body:

Request body
1{
2 "name": "get_current_weather",
3 "version_description": "Fetches current weather and uses celsius or fahrenheit based on user's location.",
4 "description": "This tool is for getting the current weather.",
5 "parameters": "{ \"type\": \"object\", \"properties\": { \"location\": { \"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\" }, \"format\": { \"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"], \"description\": \"The temperature unit to use. Infer this from the users location.\" } }, \"required\": [\"location\", \"format\"] }"
6}

The parameters field must contain a valid JSON schema.

Sample response body
1{
2 "tool_type": "FUNCTION",
3 "id": "15c38b04-ec9c-4ae2-b6bc-5603512b5d00",
4 "version": 0,
5 "version_description": "Fetches current weather and uses celsius or fahrenheit based on user's location.",
6 "name": "get_current_weather",
7 "created_on": 1714421925626,
8 "modified_on": 1714421925626,
9 "fallback_content": null,
10 "description": "This tool is for getting the current weather.",
11 "parameters": "{ \"type\": \"object\", \"properties\": { \"location\": { \"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\" }, \"format\": { \"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"], \"description\": \"The temperature unit to use. Infer this from the users location.\" } }, \"required\": [\"location\", \"format\"] }"
12}

Record the value in the id field, as we will use it to specify the newly created Tool in the next step.

Create a Configuration

Next we will create an EVI Configuration called Weather Assistant Config, and include the created Tool by making a POST request to /configs with the following request body:

Request body
1{
2 "name": "Weather Assistant Config",
3 "language_model": {
4 "model_provider": "OPEN_AI",
5 "model_resource": "gpt-3.5-turbo",
6 "temperature": null
7 },
8 "tools": [
9 {
10 "id": "15c38b04-ec9c-4ae2-b6bc-5603512b5d00",
11 "version": 0
12 }
13 ]
14}
Sample response body
1{
2 "id": "87e88a1a-3768-4a01-ba54-2e6d247a00a7",
3 "version": 0,
4 "version_description": null,
5 "name": "Weather Assistant Config",
6 "created_on": 1714421581844,
7 "modified_on": 1714421581844,
8 "prompt": null,
9 "voice": null,
10 "language_model": {
11 "model_provider": "OPEN_AI",
12 "model_resource": "gpt-3.5-turbo",
13 "temperature": null
14 },
15 "tools": [
16 {
17 "tool_type": "FUNCTION",
18 "id": "15c38b04-ec9c-4ae2-b6bc-5603512b5d00",
19 "version": 0,
20 "version_description": "Fetches current weather and uses celsius or fahrenheit based on user's location.",
21 "name": "get_current_weather",
22 "created_on": 1714421925626,
23 "modified_on": 1714421925626,
24 "fallback_content": null,
25 "description": "This tool is for getting the current weather.",
26 "parameters": "{ \"type\": \"object\", \"properties\": { \"location\": { \"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\" }, \"format\": { \"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"], \"description\": \"The temperature unit to use. Infer this from the users location.\" } }, \"required\": [\"location\", \"format\"] }"
27 }
28 ],
29 "builtin_tools": []
30}

Ensure your tool definitions conform to the language model’s schema. The specified language model will be the one to execute the function calls.

Function calling

In this section we will go over the end-to-end flow of a function call within a chat session. This flow will be predicated on having specified the Weather Assistant Config when establishing a connection with EVI. See our Configuration Guide for details on how to apply your configuration when connecting.

Currently, EVI does not support parallel function calling. Only one function call can be processed at a time.

Invoke function call

With EVI configured to use the get_current_weather Tool, we can now ask it: “what is the weather in New York?” We can expect EVI to respond with a user_message and a tool_call message:

Sample user_message
1{
2 "type": "user_message",
3 "message": {
4 "role": "user",
5 "content": "What's the weather in New York?"
6 },
7 // ...etc
8}
Sample tool_call
1{
2 "type": "tool_call",
3 "tool_type": "function",
4 "response_required": true,
5 "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
6 "name": "get_current_weather",
7 "parameters": "{\"location\":\"New York\",\"format\":\"fahrenheit\"}",
8}

Next, extract the tool_call_id from the tool_call message to be used in the next step. Then, you will need to pass the parameters from the tool_call message to your function to retrieve the weather for the designated city in the specified format.

While EVI will send a message to indicate when to invoke your function and which parameters to pass into it, you will need to define the function itself in your code. For the sake of this example, you can define a function which actually calls a weather API, or simply hard code a return value like: 60F.

Send function call result

Upon receiving the response from your function, we will then send a tool_response message containing the result. The specified tool_call_id should match the one received in the tool_call message in the previous step.

tool_response
1{
2 "type": "tool_response",
3 "tool_call_id":"call_m7PTzGxrD0i9oCHiquKIaibo",
4 "content":"60F"
5}

EVI responds

After the interface receives the tool_response message, it will then send an assistant_message containing the response generated from the reported result of the function call:

Sample assistant_message
1{
2 "type": "assistant_message",
3 "message": {
4 "role": "assistant",
5 "content": "The current weather in New York is 60F."
6 }
7}

Using built-in tools

User-defined tools allow EVI to identify when a function should be invoked, but you implement the function itself. On the other hand, Hume also provides built-in tools that are natively integrated. This means that you don’t need to define the function; EVI handles both determining when the function needs to be called and invoking it.

One such example of a built-in tool we provide is Web search. Web search equips EVI with the ability to search the web for up-to-date information.

This section explains how to specify built-in tools in your configurations and details the message flow you can expect when EVI uses a built-in tool during a chat session.

Specify built-in tool in EVI configuration

Let’s begin by creating a configuration which includes the built-in web search tool by making a POST request to /configs with the following request body:

Request body
1{
2 "name": "Web Search Config",
3 "language_model": {
4 "model_provider": "OPEN_AI",
5 "model_resource": "gpt-3.5-turbo"
6 },
7 "builtin_tools": [
8 {
9 "name": "web_search",
10 "fallback_content": "Optional fallback content to inform EVI’s spoken response if web search is not successful."
11 }
12 ]
13}
Sample response body
1{
2 "id": "3a60e85c-d04f-4eb5-8076-fb4bd344d5d0",
3 "version": 0,
4 "version_description": null,
5 "name": "Web Search Config",
6 "created_on": 1714421925626,
7 "modified_on": 1714421925626,
8 "prompt": null,
9 "voice": null,
10 "language_model": {
11 "model_provider": "OPEN_AI",
12 "model_resource": "gpt-3.5-turbo",
13 "temperature": null
14 },
15 "tools": [],
16 "builtin_tools": [
17 {
18 "tool_type": "BUILTIN",
19 "name": "web_search",
20 "fallback_content": "Optional fallback content to inform EVI’s spoken response if web search is not successful."
21 }
22 ]
23}

EVI uses built-in tool

Now that we’ve created an EVI configuration which includes the built-in web search tool, let’s review the message flow for when web search is invoked.

Web search message flow
1// 1. User asks EVI for the latest news in AI research
2{
3 "type": "user_message",
4 "message": {
5 "role": "user",
6 "content": "What is the latest news with AI research?"
7 },
8 // ...etc
9}
10// 2. EVI infers it needs to use web search, generates a search query, and invokes Hume's native web search function
11{
12 "name": "web_search",
13 "parameters": "{\"query\":\"latest news AI research\"}",
14 "tool_call_id": "call_zt1NYGpPkhR7v4kb4RPxTkLn",
15 "type": "tool_call",
16 "tool_type": "builtin",
17 "response_required": false
18}
19// 3. EVI sends back the web search results
20{
21 "type": "tool_response",
22 "tool_call_id": "call_zt1NYGpPkhR7v4kb4RPxTkLn",
23 "content": "{ \”summary\”:null, “references”: [{\”content\”:\”The latest NVIDIA news is...etc.\”, \”url\”:\”https://www.artificialintelligence-news.com/\”, \”name\”:\”AI News - Artificial Intelligence News\”}] }",
24 "tool_name": "web_search",
25 "tool_type": "builtin"
26}
27// 4. EVI sends a response generated from the web search results
28{
29 "type": "assistant_message",
30 "message": {
31 "role": "assistant",
32 "content": "IBM Research unveiled a breakthrough analog AI chip for efficient deep learning, and Quantum AI is making transformative advancements by harnessing quantum mechanics."
33 },
34 // ...etc
35}

Interruptibility

Function calls can be interrupted to cancel them or to resend them with updated parameters.

Canceling a function call

Just as EVI is able to infer when to make a function call, it can also infer from the user’s input when to cancel one. Here is an overview of what the message flow would look like:

Cancel function call message flow
1// 1. User asks what the weather is in New York
2{
3 "type": "user_message",
4 "message": {
5 "role": "user",
6 "content": "What's the weather in New York?"
7 },
8 // ...etc
9}
10// 2. EVI infers it is time to make a function call
11{
12 "type": "tool_call",
13 "tool_type": "function",
14 "response_required": true,
15 "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
16 "name": "get_current_weather",
17 "parameters": "{\"location\":\"New York\",\"format\":\"fahrenheit\"}",
18}
19// 3. User communicates sudden disinterested in the weather
20{
21 "type": "user_message",
22 "message": {
23 "role": "user",
24 "content": "Actually, never mind."
25 }
26}
27// 4. EVI infers the function call should be canceled
28{
29 "type": "assistant_message",
30 "message": {
31 "role": "assistant",
32 "content": "Okay, never mind then. Can I help you with anything else?"
33 },
34 // ...etc
35 }

Updating a function call

Sometimes we don’t necessarily want to cancel the function call, and instead want to update the parameters. EVI can infer the difference. Below is a sample flow of interrupting the interface to update the parameters of the function call:

Update function call message flow
1// 1. User asks EVI what the weather is in New York
2{
3 "type": "user_message",
4 "message": {
5 "role": "user",
6 "content": "What's the weather in New York?"
7 },
8 // ...etc
9}
10// 2. EVI infers it is time to make a function call
11{
12 "type": "tool_call",
13 "tool_type": "function",
14 "response_required": true,
15 "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
16 "name": "get_current_weather",
17 "parameters": "{\"location\":\"New York\",\"format\":\"fahrenheit\"}",
18}
19// 3. User communicates to EVI they want the weather in Los Angeles instead
20{
21 "type": "user_message",
22 "message": {
23 "role": "user",
24 "content": "Actually, Los Angeles."
25 }
26}
27// 4. EVI infers the parameters to function call should be updated
28{
29 "type": "tool_call",
30 "response_required": true,
31 "tool_call_id": "call_5RWLt3IMQyayzGdvMQVn5AOQ",
32 "name": "get_current_weather",
33 "parameters": "{\"location\":\"Los Angeles\",\"format\":\"celsius\"}",
34}
35// 5. User sends results of function call to EVI
36{
37 "type": "tool_response",
38 "tool_call_id":"call_5RWLt3IMQyayzGdvMQVn5AOQ",
39 "content":"72F"
40}
41// 6. EVI sends response container function call result
42{
43 "type": "assistant_message",
44 "message": {
45 "role": "assistant",
46 "content": "The current weather in Los Angeles is 72F."
47 },
48 // ...etc
49}

Handling errors

It’s possible for tool use to fail. For example, it can fail if the tool_response message content was not in UTF-8 format or if the function call response timed out. This section outlines how to specify fallback content to be used by EVI to communicate a failure, as well as the message flow for when a function call failure occurs.

Specifying fallback content

When defining your Tool, you can specify fallback content within the Tool’s fallback_content field. When the Tool fails to generate content, the text in this field will be sent to the LLM in place of a result. To accomplish this, let’s update the Tool we created during setup to include fallback content. We can accomplish this by publishing a new version of the Tool via a POST request to /tools/{id}:

Request body
1{
2 "version_description": "Adds fallback content",
3 "description": "This tool is for getting the current weather.",
4 "parameters": "{ \"type\": \"object\", \"properties\": { \"location\": { \"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\" }, \"format\": { \"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"], \"description\": \"The temperature unit to use. Infer this from the users location.\" } }, \"required\": [\"location\", \"format\"] }",
5 "fallback_content": "Something went wrong. Failed to get the weather."
6}
Sample response body
1{
2 "tool_type": "FUNCTION",
3 "id": "36f09fdc-4630-40c0-8afa-6a3bdc4eb4b1",
4 "version": 1,
5 "version_type": "FIXED",
6 "version_description": "Adds fallback content",
7 "name": "get_current_weather",
8 "created_on": 1714421925626,
9 "modified_on": 1714425632084,
10 "fallback_content": "Something went wrong. Failed to get the weather.",
11 "description": null,
12 "parameters": "{ \"type\": \"object\", \"properties\": { \"location\": { \"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\" }, \"format\": { \"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"], \"description\": \"The temperature unit to use. Infer this from the user's location.\" } }, \"required\": [\"location\", \"format\"] }"
13}

Failure message flow

This section outlines the sort of messages that can be expected when Tool use fails. After sending a tool-response message, we will know an error, or failure, occurred when we receive the tool_error message:

Bad function call response error flow
1// 1. User asks EVI what the weather is in New York
2{
3 "type": "user_message",
4 "message": {
5 "role": "user",
6 "content": "What's the weather in New York?"
7 },
8 // ...etc
9}
10// 2. EVI infers it is time to make a function call
11{
12 "type": "tool_call",
13 "tool_type": "function",
14 "response_required": true,
15 "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
16 "name": "get_current_weather",
17 "parameters": "{\"location\":\"New York\",\"format\":\"fahrenheit\"}",
18}
19// 3. User sends results of function call to EVI (result not formatted correctly)
20{
21 "type": "tool_response",
22 "tool_call_id":"call_5RWLt3IMQyayzGdvMQVn5AOQ",
23 "content":"60F"
24}
25// 4. EVI sends response communicating it failed to process the tool_response
26{
27 "type": "tool_error",
28 "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
29 "error": "Malformed tool response: <error message here>",
30 "fallback_content": "Something went wrong. Failed to get the weather.",
31 "level": "warn"
32}
33// 5. EVI generates a response based on the failure
34{
35 "type": "assistant_message",
36 "message": {
37 "role": "assistant",
38 "content": "Sorry, I wasn't able to get the weather. Can I help with anything else?"
39 },
40 // ...etc
41}

Let’s cover another type of failure scenario: what if the weather API the function was using was down? In this case we would send EVI a tool_error message. When sending the tool_error message we can specify fallback_content more specific to the error our function throws. This is what the message flow would be for this type of failure:

Failed function call flow
1// 1. User asks EVI what the weather is in New York
2{
3 "type": "user_message",
4 "message": {
5 "role": "user",
6 "content": "What's the weather in New York?"
7 },
8 // ...etc
9}
10// 2. EVI infers it is time to make a function call
11{
12 "type": "tool_call",
13 "tool_type": "function",
14 "response_required": true,
15 "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
16 "name": "get_current_weather",
17 "parameters": "{\"location\":\"New York\",\"format\":\"fahrenheit\"}",
18}
19// 3. Function failed, so we send EVI a message communicating the failure on our end
20{
21 "type": "tool_error",
22 "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
23 "error": "Malformed tool response: <error message here>",
24 "fallback_content": "Function execution failure - weather API down.",
25 "level": "warn"
26}
27// 4. EVI generates a response based on the failure
28{
29 "type": "assistant_message",
30 "message": {
31 "role": "assistant",
32 "content": "Sorry, our weather resource is unavailable. Can I help with anything else?"
33 },
34 // ...etc
35}