OpenAI Function Calling

This past week, OpenAI added function calling to their SDK. This addition is exciting because it now incorporates schema as a first-class citizen in making calls to OpenAI chat models. As the example code and naming suggest, you can define a list of functions and schema of the parameters required to call them and the model will determine whether a function needs to be invoked in the context of the completion, then return JSON adhering to the schema defined for the function. If you read anything else I've written you probably know what I'm going to try and do next: let's use a function to extract structured data from an unstructured input. Extract a recipe as structured data I found this recipe and I want to try it out. I want to parse the content on the page and extract the recipe in a form that I could easily render on a personal recipe site. I quickly checked the page and it looks like most of the content is nested within an html element with the class "content". Here is some Python code to extract all the text from the HTML, eliminating the markup: This code outputs a big block of text, a lot of which isn't ingredients or instructions for the recipe. Before we get into calling the language model, let's write a schema for the data we'd like to extract from the page's content. We'll use pydantic because it can easily be converted to a JSON schema. Nothing too surprising so far. Now is the interesting part. Let's wire up a call to OpenAI that uses functions and our Recipe schema to structure the response: Re-writing with a bit of refactor: Here is where I started to run into problems. When running the script, I get the following error: Inspecting the JSON output of the script, we see the model isn't returning valid JSON: We see "quantity": 3/4 isn't valid JSON. We can try to steer the model adding a description to the pydantic field: This modifies the JSON schema in the following way: Unfortunately, this doesn't resolve the invalid JSON issue. However, switching from gpt-3.5-turbo-16k to gpt-4-0613 (and removing the Field description) yields JSON that adheres to the input schema. Still, GPT-4 models are slower and more expensive than 3.5 models, so there is motivation to try and get this working with the latter. Taking an approach I've tried previously, it seems like we can get more reliable results with gpt-3.5-turbo-16k. Takeaways On one hand, it's great to see OpenAI training models to better integrate with emerging language model use cases like function invocation and schema extraction. On the other, OpenAI acknowledges this approach doesn't always work in their documentation: > the model may generate invalid JSON or hallucinate parameters Previous techniques I've explored for schema extraction seem to produce more consistent results, even with less advanced models.