Tool use

Guide to using function calling with the Empathic Voice Interface (EVI).

EVI simplifies the integration of external APIs through function calling. Developers can integrate custom functions that are invoked dynamically based on the user’s input, enabling more useful conversations. There are two key concepts for using function calling with EVI: Tools and Configurations (Configs):

  • Tools are resources that EVI uses to do things, like search the web or call external APIs. For example, tools can check the weather, update databases, schedule appointments, or take actions based on what occurs in the conversation. While the tools can be user-defined, Hume also offers natively implemented tools, like web search, which are labeled as “built-in” tools.

  • Configurations enable developers to customize an EVI’s behavior and incorporate these custom tools. Setting up an EVI configuration allows developers to seamlessly integrate their tools into the voice interface. A configuration includes prompts, user-defined tools, and other settings.

Tool use flow diagram

Currently, our function calling feature only supports OpenAI and Anthropic models. For the best results, we suggest choosing a fast and intelligent LLM that performs well on function calling benchmarks. On account of its speed and intelligence, we recommend Claude 3.5 Haiku as the supplemental LLM in your EVI configuration when using tools. Function calling is not available if you are using your own custom language model. We plan to support more function calling LLMs in the future.

The focus of this guide is on creating a Tool and a Configuration that allows EVI to use the Tool. Additionally, this guide details the message flow of function calls within a session, and outlines the expected responses when function calls fail. Refer to our Configuration Guide for detailed, step-by-step instructions on how to create and use an EVI Configuration.

Explore these sample projects to see how Tool use can be implemented in TypeScript, Next.js, and Python.

Setup

For EVI to leverage tools or call functions, a configuration must be created with the tool’s definition. Our step-by-step guide below walks you through creating a tool and adding it to a configuration, using either a no-code approach through our Portal or a full-code approach through our API.

1

Create a Tool

We will first create a Tool with a specified function. In this example, we will create a tool for getting the weather. In the Portal, navigate to the EVI Tools page. Click the Create tool button to begin.

EVI Tools page
2

Fill in Tool details

Next, we will fill in the details for a weather tool named get_current_weather. This tool fetches the current weather conditions in a specified location and reports the temperature in either Celsius or Fahrenheit. We can establish the tool’s behavior by completing the following fields:

  • Name: Specify the name of the function that the language model will invoke. Ensure it begins with a lowercase letter and only contains letters, numbers, or underscores.
  • Description: Provide a brief description of what the function does.
  • Parameters: Define the function’s input parameters using a JSON schema.
EVI Create function interface

The JSON schema defines the expected structure of a function’s input parameters. Here’s an example JSON schema we can use for the parameters field of a weather function:

parameters
1{
2  "type": "object",
3  "required": ["location", "format"],
4  "properties": {
5    "location": {
6      "type": "string",
7      "description": "The city and state, e.g. San Francisco, CA"
8    },
9    "format": {
10      "type": "string",
11      "enum": ["celsius", "fahrenheit"],
12      "description": "The temperature unit to use. Infer this from the user's location."
13    }
14  }
15}
3

Create a Configuration

Next, we will create an EVI Configuration called Weather Assistant Config. This configuration will utilize the get_current_weather Tool created in the previous step. See our Configuration guide for step-by-step instructions on how to create a configuration. During the Set up LLM step, remember to select an Anthropic or OpenAI model for tool use support.

Create a configuration called Weather Assistant Config in the Hume portal
4

Add Tool to Configuration

Finally, we will specify the get_current_weather Tool in the Weather Assistant Config. Navigate to the Tools section of the EVI Config details page. Click the Add button to add a function to your configuration. Since we have already created a get_current_weather Tool in previous steps, we can simply select Add existing tool… from the dropdown to specify it.

Add tool to configuration within the Hume portal

Select the tool to add get_current_weather to your configuration, then complete the remaining steps to create the configuration.

Add get_current_weather to configuration within the Hume portal

Function calling

In this section, we will go over the end-to-end flow of a function call within a chat session. This flow will be predicated on having specified the Weather Assistant Config when establishing a connection with EVI. See our Configuration Guide for details on how to apply your configuration when connecting.

Check out the TypeScript and Python example projects for complete implementations of the weather Tool you’ll build in this tutorial.

1

Define a function

We must first define a function for your Tool. This function will take the same parameters as those specified during your Tool’s creation.

For this tutorial, we will define a function that calls a weather API (e.g., the Geocoding API) to retrieve the weather for a designated city in a specified format. This weather function will accept location and format as its parameters.

See the code below for a sample implementation:

1async function fetchWeather(location: string, format: string): Promise<string> {
2  // Fetch the location's geographic coordinates using Geocoding API
3  const locationApiURL = `https://geocode.maps.co/search?q=${location}&api_key=${YOUR_WEATHER_API_KEY}`;
4  const locationResponse = await fetch(locationApiURL);
5  const locationData = await locationResponse.json();
6
7  // Extract latitude and longitude from fetched location data
8  const { lat, lon } = locationData[0];
9
10  // Fetch point metadata using the extracted location coordinates
11  const pointMetadataEndpoint = `https://api.weather.gov/points/${parseFloat(
12    lat
13  ).toFixed(3)},${parseFloat(lon).toFixed(3)}`;
14  const pointMetadataResponse = await fetch(pointMetadataEndpoint);
15  const pointMetadata = await pointMetadataResponse.json();
16
17  // Extract weather forecast URL from point metadata
18  const forecastUrl = pointMetadata.properties.forecast;
19
20  // Fetch the weather forecast using the forecast URL
21  const forecastResponse = await fetch(forecastUrl);
22  const forecastData = await forecastResponse.json();
23  const forecast = JSON.stringify(forecastData.properties.periods);
24
25  // Return the temperature in the specified format
26  return `${forecast} in ${format}`;
27}

Instead of calling a weather API, you can hard code a return value like 75F as a means to quickly test for the sake of this tutorial.

2

EVI signals function call

Once EVI is configured with your Tool, it will automatically infer when to signal a function call within a chat session. With EVI configured to use the get_current_weather Tool, we can now ask it: “what is the weather in New York?”

Let’s try it out in the EVI Playground.

Ask EVI what is the weather in New York

We can expect EVI to respond with a User Message and a Tool Call message:

Sample User Message
1{
2  "type": "user_message",
3  "message": {
4    "role": "user",
5    "content": "What's the weather in New York?"
6  },
7  // ...etc
8}
Sample Tool Call message
1{
2  "type": "tool_call",
3  "tool_type": "function",
4  "response_required": true,
5  "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
6  "name": "get_current_weather",
7  "parameters": "{\"location\":\"New York\",\"format\":\"fahrenheit\"}"
8}

Currently, EVI does not support parallel function calling. Only one function call can be processed at a time.

3

Extract arguments from Tool Call message

Upon receiving a Tool Call message from EVI, we will parse the parameters and extract the arguments.

The code below demonstrates how to extract the location and format arguments, which the user-defined fetch weather function is expecting, from a received Tool Call message.

1import { Hume } from 'hume';
2
3async function handleToolCallMessage(
4  toolCallMessage: Hume.empathicVoice.ToolCallMessage,
5  socket: Hume.empathicVoice.chat.ChatSocket): Promise<void> {
6  if (toolCallMessage.name === "get_current_weather") {
7    // 1. Parse the parameters from the Tool Call message
8    const args = JSON.parse(toolCallMessage.parameters) as {
9      location: string;
10      format: string;
11    };
12    // 2. Extract the individual arguments
13    const { location, format } = args;
14    // ...etc.
15  }
16}
4

Invoke function call

Next, we will pass the extracted arguments into the previously defined fetch weather function. We will capture the return value to send back to EVI:

1import { Hume } from 'hume';
2
3async function handleToolCallMessage(
4  toolCallMessage: Hume.empathicVoice.ToolCallMessage,
5  socket: Hume.empathicVoice.chat.ChatSocket): Promise<void> {
6  if (toolCallMessage.name === "get_current_weather") {
7    // 1. Parse the parameters from the Tool Call message
8    const args = JSON.parse(toolCallMessage.parameters) as {
9      location: string;
10      format: string;
11    };
12    // 2. Extract the individual arguments
13    const { location, format } = args;
14    // 3. Call fetch weather function with extracted arguments
15    const weather = await fetchWeather(location, format);
16    // ...etc.
17  }
18}
5

Send function call result

Upon receiving the return value of your function, we will send a Tool Response message containing the result. The specified tool_call_id must match the one received in the Tool Call message from EVI:

1import { Hume } from 'hume';
2
3async function handleToolCallMessage(
4  toolCallMessage: Hume.empathicVoice.ToolCallMessage,
5  socket: Hume.empathicVoice.chat.ChatSocket): Promise<void> {
6  if (toolCallMessage.name === "get_current_weather") {
7    // 1. Parse the parameters from the Tool Call message
8    const args = JSON.parse(toolCallMessage.parameters) as {
9      location: string;
10      format: string;
11    };
12    // 2. Extract the individual arguments
13    const { location, format } = args;
14    // 3. Call fetch weather function with extracted arguments
15    const weather = await fetchWeather(location, format);
16    // 4. Construct a Tool Response message containing the result
17    const toolResponseMessage = {
18      type: "tool_response",
19      toolCallId: toolCallMessage.toolCallId,
20      content: weather,
21    };
22    // 5. Send Tool Response message to the WebSocket
23    socket.sendToolResponseMessage(toolResponseMessage);
24  }
25}

Let’s try it in the EVI Playground. Enter the return value of your function in the input field below the Tool Call message, and click Send Response. In practice, you will use the actual return value from your function call. However, for demonstration purposes, we will assume a return value of “75F”.

Send EVI function result
6

EVI responds

After the interface receives the Tool Response message, it will then send an Assistant Message containing the response generated from the reported result of the function call:

Sample assistant_message
1{
2  "type": "assistant_message",
3  "message": {
4    "role": "assistant",
5    "content": "The current temperature in New York, NY is 75F."
6  }
7}

See how it works in the EVI Playground.

EVI responds with function call result

To summarize, Tool Call serves as a programmatic tool for intelligently signaling when you should invoke your function. EVI does not invoke the function for you. You will need to define a function, invoke the function, and pass the return value of your function to EVI via a Tool Response message. EVI will generate a response based on the content of your message.

Using built-in tools

User-defined tools allow EVI to identify when a function should be invoked, but you will need to invoke the function itself. On the other hand, Hume also provides built-in tools that are natively integrated. This means that you don’t need to define the function; EVI handles both determining when the function needs to be called and invoking it.

Hume supports the following built-in tools:

  • web_search: Enables EVI to search the web for real-time information when needed.
  • hang_up: Closes the WebSocket connection with status code 1000 when appropriate (e.g., after detecting a farewell, signaling the end of the conversation).

This section explains how to specify built-in tools in your configurations and details the message flow you can expect when EVI uses a built-in tool during a chat session.

1

Specify built-in tool in EVI configuration

Let’s begin by creating a configuration which includes the built-in web search tool. To specify the web search tool in your EVI configuration, during the Add tools step, ensure Web search is enabled. Refer to our Configuration Guide for more details on creating a configuration.

Create a configuration with a built-in web search tool

Alternatively, you can specify the built-in tool by making a POST request to /configs with the following request body:

Request body
1{
2  "name": "Web Search Config",
3  "language_model": {
4    "model_provider": "OPEN_AI",
5    "model_resource": "gpt-3.5-turbo"
6  },
7  "builtin_tools": [
8    { 
9      "name": "web_search",
10      "fallback_content": "Optional fallback content to inform EVI’s spoken response if web search is not successful."
11    }
12  ]
13}

Upon success, expect EVI to return a response similar to this example:

Sample response body
1{
2  "id": "3a60e85c-d04f-4eb5-8076-fb4bd344d5d0",
3  "version": 0,
4  "version_description": null,
5  "name": "Web Search Config",
6  "created_on": 1714421925626,
7  "modified_on": 1714421925626,
8  "prompt": null,
9  "voice": null,
10  "language_model": {
11    "model_provider": "OPEN_AI",
12    "model_resource": "gpt-3.5-turbo",
13    "temperature": null
14  },
15  "tools": [],
16  "builtin_tools": [
17    {
18      "tool_type": "BUILTIN",
19      "name": "web_search",
20      "fallback_content": "Optional fallback content to inform EVI’s spoken response if web search is not successful."
21    }
22  ]
23}
2

EVI uses built-in tool

Now that we’ve created an EVI configuration which includes the built-in web search tool, let’s test it out in the EVI Playground. Try asking EVI a question that requires web search, like “what is the latest news with AI research?”

Ask EVI what is the latest news with AI research

EVI will send a response generated from the web search results:

EVI sends a response generated from web search results

Let’s review the message flow for when web search is invoked.

Web search message flow
1// 1. User asks EVI for the latest news in AI research
2{
3  "type": "user_message",
4  "message": {
5    "role": "user",
6    "content": "What is the latest news with AI research?"
7  },
8  // ...etc
9}
10// 2. EVI infers it needs to use web search, generates a search query, and invokes Hume's native web search function
11{
12  "name": "web_search", 
13  "parameters": "{\"query\":\"latest news AI research\"}", 
14  "tool_call_id": "call_zt1NYGpPkhR7v4kb4RPxTkLn", 
15  "type": "tool_call", 
16  "tool_type": "builtin", 
17  "response_required": false
18}
19// 3. EVI sends back the web search results 
20{
21  "type": "tool_response", 
22  "tool_call_id": "call_zt1NYGpPkhR7v4kb4RPxTkLn", 
23  "content": "{ \”summary\”:null, “references”: [{\”content\”:\”Researchers have demonstrated a new method...etc.\”, \”url\”:\”https://www.sciencedaily.com/news/computers_math/artificial_intelligence/\”, \”name\”:\”Artificial Intelligence News -- ScienceDaily\”}] }", 
24  "tool_name": "web_search", 
25  "tool_type": "builtin"
26}
27// 4. EVI sends a response generated from the web search results
28{
29  "type": "assistant_message", 
30  "message": {
31    "role": "assistant", 
32    "content": "Oh, there's some interesting stuff happening in AI research right now."
33  },
34  // ...etc
35}
36{
37  "type": "assistant_message", 
38  "message": {
39    "role": "assistant", 
40    "content": "Just a few hours ago, researchers demonstrated a new method using AI and computer simulations to train robotic exoskeletons."
41  },
42  // ...etc
43}

Interruptibility

Function calls can be interrupted to cancel them or to resend them with updated parameters.

Canceling a function call

Just as EVI is able to infer when to make a function call, it can also infer from the user’s input when to cancel one. Here is an overview of what the message flow would look like:

User signals they want to cancel a function call
EVI infers from user input to cancel function call
Cancel function call message flow
1// 1. User asks what the weather is in New York
2{
3  "type": "user_message",
4  "message": {
5    "role": "user",
6    "content": "What's the weather in New York?"
7  },
8  // ...etc
9}
10// 2. EVI infers it is time to make a function call
11{
12  "type": "tool_call",
13  "tool_type": "function",
14  "response_required": true,
15  "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
16  "name": "get_current_weather",
17  "parameters": "{\"location\":\"New York\",\"format\":\"fahrenheit\"}"
18}
19// 3. User communicates sudden disinterested in the weather
20{
21  "type": "user_message",
22  "message": {
23    "role": "user",
24    "content": "Actually, never mind."
25  }
26}
27// 4. EVI infers the function call should be canceled
28{
29    "type": "assistant_message",
30    "message": {
31      "role": "assistant",
32      "content": "If you change your mind or need any weather information in the future, feel free to let me know."
33    },
34    // ...etc
35  }

Updating a function call

Sometimes we don’t necessarily want to cancel the function call, and instead want to update the parameters. EVI can infer the difference. Below is a sample flow of interrupting the interface to update the parameters of the function call:

User asks EVI the weather in New York
EVI updates function call to get weather in Los Angeles
Update function call message flow
1// 1. User asks EVI what the weather is in New York
2{
3  "type": "user_message",
4  "message": {
5    "role": "user",
6    "content": "What's the weather in New York?"
7  },
8  // ...etc
9}
10// 2. EVI infers it is time to make a function call
11{
12  "type": "tool_call",
13  "tool_type": "function",
14  "response_required": true,
15  "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
16  "name": "get_current_weather",
17  "parameters": "{\"location\":\"New York\",\"format\":\"fahrenheit\"}"
18}
19// 3. User communicates to EVI they want the weather in Los Angeles instead
20{
21  "type": "user_message",
22  "message": {
23    "role": "user",
24    "content": "Actually, Los Angeles."
25  }
26}
27// 4. EVI infers the parameters to function call should be updated
28{
29  "type": "tool_call",
30  "response_required": true,
31  "tool_call_id": "call_5RWLt3IMQyayzGdvMQVn5AOQ",
32  "name": "get_current_weather",
33  "parameters": "{\"location\":\"Los Angeles\",\"format\":\"celsius\"}"
34}
35// 5. User sends results of function call to EVI
36{
37  "type": "tool_response",
38  "tool_call_id":"call_5RWLt3IMQyayzGdvMQVn5AOQ",
39  "content":"72F"
40}
41// 6. EVI sends response container function call result
42{
43  "type": "assistant_message",
44  "message": {
45    "role": "assistant",
46    "content": "The current weather in Los Angeles is 72F."
47  },
48  // ...etc
49}

Handling errors

It’s possible for tool use to fail. For example, it can fail if the Tool Response message content was not in UTF-8 format or if the function call response timed out. This section outlines how to specify fallback content to be used by EVI to communicate a failure, as well as the message flow for when a function call failure occurs.

Specifying fallback content

When defining your Tool, you can specify fallback content within the Tool’s fallback_content field. When the Tool fails to generate content, the text in this field will be sent to the LLM in place of a result. To accomplish this, let’s update the Tool we created during setup to include fallback content. We can accomplish this by publishing a new version of the Tool via a POST request to /tools/{id}:

Request body
1{
2  "version_description": "Adds fallback content",
3  "description": "This tool is for getting the current weather.",
4  "parameters": "{ \"type\": \"object\", \"properties\": { \"location\": { \"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\" }, \"format\": { \"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"], \"description\": \"The temperature unit to use. Infer this from the users location.\" } }, \"required\": [\"location\", \"format\"] }",
5  "fallback_content": "Something went wrong. Failed to get the weather."
6}
Sample response body
1{
2  "tool_type": "FUNCTION",
3  "id": "36f09fdc-4630-40c0-8afa-6a3bdc4eb4b1",
4  "version": 1,
5  "version_type": "FIXED",
6  "version_description": "Adds fallback content",
7  "name": "get_current_weather",
8  "created_on": 1714421925626,
9  "modified_on": 1714425632084,
10  "fallback_content": "Something went wrong. Failed to get the weather.",
11  "description": null,
12  "parameters": "{ \"type\": \"object\", \"properties\": { \"location\": { \"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\" }, \"format\": { \"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"], \"description\": \"The temperature unit to use. Infer this from the user's location.\" } }, \"required\": [\"location\", \"format\"] }"
13}

Failure message flow

This section outlines the sort of messages that can be expected when Tool use fails. After sending a Tool Response message, we will know an error, or failure, occurred when we receive the Tool Error message:

EVI responds with a tool_error message
Bad function call response error flow
1// 1. User asks EVI what the weather is in New York
2{
3  "type": "user_message",
4  "message": {
5    "role": "user",
6    "content": "What's the weather in New York?"
7  },
8  // ...etc
9}
10// 2. EVI infers it is time to make a function call
11{
12  "type": "tool_call",
13  "tool_type": "function",
14  "response_required": true,
15  "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
16  "name": "get_current_weather",
17  "parameters": "{\"location\":\"New York\",\"format\":\"fahrenheit\"}"
18}
19// 3. User sends results of function call to EVI (result not formatted correctly)
20{
21  "type": "tool_response",
22  "tool_call_id":"call_5RWLt3IMQyayzGdvMQVn5AOQ",
23  "content":"MALFORMED RESPONSE"
24}
25// 4. EVI sends response communicating it failed to process the tool_response
26{
27  "type": "tool_error",
28  "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
29  "error": "Malformed tool response: <error message here>",
30  "fallback_content": "Something went wrong. Failed to get the weather.",
31  "level": "warn"
32}
33// 5. EVI generates a response based on the failure
34{
35  "type": "assistant_message",
36  "message": {
37    "role": "assistant",
38    "content": "It looks like there was an issue retrieving the weather information for New York."
39  },
40  // ...etc
41}

Let’s cover another type of failure scenario: what if the weather API the function was using was down? In this case, we would send EVI a Tool Error message. When sending the Tool Error message, we can specify fallback_content to be more specific to the error our function throws. This is what the message flow would be for this type of failure:

Failed function call flow
1// 1. User asks EVI what the weather is in New York
2{
3  "type": "user_message",
4  "message": {
5    "role": "user",
6    "content": "What's the weather in New York?"
7  },
8  // ...etc
9}
10// 2. EVI infers it is time to make a function call
11{
12  "type": "tool_call",
13  "tool_type": "function",
14  "response_required": true,
15  "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
16  "name": "get_current_weather",
17  "parameters": "{\"location\":\"New York\",\"format\":\"fahrenheit\"}"
18}
19// 3. Function failed, so we send EVI a message communicating the failure on our end
20{
21  "type": "tool_error",
22  "tool_call_id": "call_m7PTzGxrD0i9oCHiquKIaibo",
23  "error": "Malformed tool response: <error message here>",
24  "fallback_content": "Function execution failure - weather API down.",
25  "level": "warn"
26}
27// 4. EVI generates a response based on the failure
28{
29  "type": "assistant_message",
30  "message": {
31    "role": "assistant",
32    "content": "Sorry, our weather resource is unavailable. Can I help with anything else?"
33  },
34  // ...etc
35}

Let’s revisit our function for handling Tool Call messages from the Function Calling section. We can now add support for error handling by sending Tool Error messages to EVI. This will enable our function to handle cases where fetching the weather fails or the requested tool is not found:

1import { Hume } from 'hume';
2
3async function handleToolCallMessage(
4  toolCallMessage: Hume.empathicVoice.ToolCallMessage,
5  socket: Hume.empathicVoice.chat.ChatSocket): Promise<void> {
6  if (toolCallMessage.name === "get_current_weather") {
7    try{
8      // parse the parameters from the Tool Call message
9      const args = JSON.parse(toolCallMessage.parameters) as {
10        location: string;
11        format: string;
12      };
13      // extract the individual arguments
14      const { location, format } = args;
15      // call fetch weather function with extracted arguments
16      const weather = await fetchWeather(location, format);
17      // send Tool Response message to the WebSocket
18      const toolResponseMessage = {
19        type: "tool_response",
20        toolCallId: toolCallMessage.toolCallId,
21        content: weather,
22      };
23      socket.sendToolResponseMessage(toolResponseMessage);
24    } catch (error) {
25      // send Tool Error message if weather fetching fails
26      const weatherToolErrorMessage = {
27        type: "tool_error",
28        toolCallId: toolCallMessage.toolCallId,
29        error: "Weather tool error",
30        content: "There was an error with the weather tool",
31      };
32      socket.sendToolErrorMessage(weatherToolErrorMessage);
33    }
34  } else {
35    // send Tool Error message if the requested tool was not found
36    const toolNotFoundErrorMessage = {
37      type: "tool_error",
38      toolCallId: toolCallMessage.toolCallId,
39      error: "Tool not found",
40      content: "The tool you requested was not found",
41    };
42    socket.sendToolErrorMessage(toolNotFoundErrorMessage);
43  }
44}