Promptfoo and standardizing output structure across models
promptfoo is a Javascript library and CLI for testing and evaluating LLM output quality. It's straightforward to install and get up and running quickly. As a first experiment, I've used it to compare the output of three similar prompts that specify their output structure using different modes of schema definition. To get started
The scaffold creates a prompts.txt file, and this is where I wrote a parameterized prompt to classify and extract data from a support message.
The more interesting part is the promptfooconfig.yaml file which specifies the different inputs and schemas:
Since we're keeping input constant, the output is a matrix of result that vary by schema and provider.
| Provider | Schema | Output |
|---|---|---|
| gpt-3.5-turbo-16k | Pydantic | { "message_type": "complaint" } |
| gpt-3.5-turbo-16k | JSON | { "message_type": "complaint" } |
| gpt-3.5-turbo-16k | Protobuf | { "order_id": "", "message_type": "COMPLAINT" } |
| gpt-4 | Pydantic | { "order_id": null, "message_type": "complaint" } |
| gpt-4 | JSON | { "order_id": null, "message_type": "complaint" } |
| gpt-4 | Protobuf | { "order_id": "", "message_type": 1 } |
This is a lot of variety! However, the results could be helpful as we think about the output we actually want and the model we plan to use. We see order_id is sometimes omitted, sometimes the empty string "" and sometimes null. We also see message_type shows up as the upper and lowercase version of "complaint" as well as the corresponding protobuf enum integer value 1.
It would be nice if there was more consistency.
It would be useful (if possible) to write prompts that yield object schema with consistent structure across models. We could try and prompt engineer in this direction or try and create a dataset and fine tune.
Some minor changes to the prompt yield consistency across all schemas and models for the input.
With the new prompt, all outputs become
Discussion in the ATmosphere