VLMs Hallucinate

import Chat from '@components/prose/Chat.astro'; import { Tweet } from 'astro-embed'; import vlmsHallucinateImage from './images/vlms-hallucinate.png';

I've done some experimentation extracting structured data from documents using VLMs. A summary of one approach I've tried can be found in my repo, impulse. I've found using Protobufs to be a relatively effective approach for extracting values from documents. The high-level idea is you write a Protobuf as your target data model then use that Protobuf itself as most of the prompt[^1]. I discussed the approach in more detail in this post so I am going to jump right into it.

The problem

When relevant contextual data is available in an image, it can be hard to prevent a VLM from hallucinating values that plausibly could be in the image, even when they're not. For reference, the image below is a png version of receipt-no-tax-or-totals.pdf which I will reference later. The data is fake, generated by a model to look realistic. The example:

<Chat model="gpt-4o-2024-08-06" messages={[ { role: 'user', content: [ { type: 'text', text: '\n\nUsing the provided content and images, extract an instance of Receipt as JSON in adherence to the above schema.\nNo talk. JSON only.', }, { type: 'image_url', image_url: { url: vlmsHallucinateImage.src, }, }, ], }, { role: 'assistant', content: [ { type: 'text', text: '', }, ], }, ]} />

Looking at the model output, we see subtotal, tax and total in the response.

claude-3.5-sonnet has the same challenges.

<Chat model="claude-3-5-sonnet-20240620" messages={[ { role: 'user', content: [ { type: 'text', text: '\n\nUsing the provided content and images, extract an instance of Receipt as JSON in adherence to the above schema.\nNo talk. JSON only.', }, { type: 'image_url', image_url: { url: vlmsHallucinateImage.src, }, }, ], }, { role: 'assistant', content: [ { type: 'text', text: '', }, ], }, ]} />

Why?

In the Protobuf, we specified these fields as optional, yet the model has output them anyway[^2]. This doesn't happen every time but it happens far more often than I would like.

We could handwave these results away and say more prompt engineering would help but it doesn't seem like it does, at least not reliably.

It's been difficult to find stability to this particular extraction. There are a lot of elements that can be varied.

• the schema (Protobuf, Pydantic, JSON schema, etc.)
• the surrounding prompt, e.g. "... No talk. JSON only".
• the image quality
• whether the image has the subtotal, tax and total labels (with the values always missing)

I tried 4 different Protobufs against 5 different receipt PDFs. receipt-original.pdf was a standard receipt with all the data you would expect. Both models consistently extract the data correctly from these -- 8/8 tests were successful.

As soon as I removed the values for subtotal, tax and total, we start seeing the hallucinations. I tried examples with just the values removed and with the values and labels removed. We see test failures (hallucinations) by both models across all of these examples.

For receipt-no-tax-or-totals.pdf, the receipt with the subtotal, tax and total labels but values missing, 5/8 of the tests fail, meaning the models outputted at least one of these values even though they aren't actually in the document.

I ran three more rounds of testing for this document specifically.

In all but one test, we see hallucinations at least 1/3 times and the only approach for which we don't (claude-3-5-sonnet-20240620/receipt_item_comments.proto) is pretty kludgy.

Takeaways

There are lots more things to try her