{
"path": "/posts/2025/document-citations/index",
"site": "at://did:plc:mracrip6qu3vw46nbewg44sm/site.standard.publication/self",
"$type": "site.standard.document",
"title": "Document Citations",
"updatedAt": "2025-02-02T03:39:49.000Z",
"description": "Using Claude Citations to annotate the sources for document Q&A",
"publishedAt": "2025-02-02T03:39:49.000Z",
"textContent": "The following is an implementation of document Q&A with citations using Anthropic's press release for Claude Citations, a feature meant to reduce LLM hallucinations and provide clarity for which part of the source text the model is using to produce answers to questions.\nThe content below the next page break is from Anthropic (the source document in our example).\nBelow are example questions that can be asked about the document.\nWhen provided with the document as context, an LLM can generate answers to these questions.\n\nTry it out\n\nSelect a question to view the model-generated answer using the source document.\nDoing so will also highlight the areas of the document that Claude returned as citations, justifying the answer.\nIf you scroll to the bottom of an answer, you can also find links to the cited passages.\n\n---\n\nexport const qaPairs = [\n {\n question: 'Does using the Citations feature improve recall accuracy?',\n answer:\n \"Yes, according to the documents, the Citations feature does improve recall accuracy. Specifically, Anthropic's internal evaluations show that Claude's built-in citation capabilities outperform most custom implementations, increasing recall accuracy by up to 15%.\\n\\nThis improvement is further supported by real-world customer experiences. For example, Endex reported that using Anthropic's Citations feature reduced source hallucinations and formatting issues from 10% to 0% and led to a 20% increase in references per response.\\n\\nThe improvement comes from replacing the previous approach, where developers had to rely on complex prompts to instruct Claude to include source information - a method that resulted in inconsistent performance and required significant time investment in prompt engineering and testing. With Citations, users can simply add source documents to the context window, and Claude automatically cites claims from those sources.\",\n citations: [\n {\n text: \"Our internal evaluations show that Claude's built-in citation capabilities outperform most custom implementations, increasing recall accuracy by up to 15%.1\\n\\nUse cases\\n\\nWith Citations, developers can create AI solutions that offer enhanced accountability across use cases like:\\n\\n Document summarization: Generate concise summaries of long documents, like case files, with each key point linked back to its original source.\",\n start_index: 1041,\n end_index: 1474,\n document: 'Citations on the Anthropic API',\n },\n {\n text: '\"With Anthropic\\'s Citations, we reduced source hallucinations and formatting issues from 10% to 0% and saw a 20% increase in references per response.',\n start_index: 3776,\n end_index: 3926,\n document: 'Citations on the Anthropic API',\n },\n {\n text: 'Previously, developers relied on complex prompts that instruct Claude to include source information, often resulting in inconsistent performance and significant time investment in prompt engineering and testing. With Citations, users can now add source documents to the context window, and when querying the model, Claude automatically cites claims in its output that are inferred from those sources.',\n start_index: 639,\n end_index: 1041,\n document: 'Citations on the Anthropic API',\n },\n ],\n },\n {\n question: 'How does the Citations feature work technically?',\n answer:\n \"Based on the documents, here's how the Citations feature works technically:\\n\\nThe process begins when the API receives user-provided source documents (which can be PDF documents or plain text files). These documents are first processed by chunking them into sentences.\\n\\nAfter chunking, these sentences are passed to the model along with:\\n- The user's query\\n- Any user-provided context\\nNote that users have the flexibility to provide their own chunks for the source documents if they prefer.\\n\\nWhen generating a response, Claude analyzes the query and provides precise citations for any claims that are derived from the source material. The citation system is designed to reference source documents in a way that helps minimize hallucinations.\\n\\nA key technical advantage of this approach is its flexibility and ease of use - it doesn't require file storage and integrates seamlessly with the Messages API.\\n\\nRegarding performance and pricing:\\n\\nInternal evaluations have shown that Claude's built-in citation capabilities perform better than most custom implementations, with up to 15% higher recall accuracy.\\n\\nThe feature uses Anthropic's standard token-based pricing model. While processing documents may require additional input tokens, users are not charged for output tokens that return the quoted text itself.\",\n citations: [\n {\n text: 'How it works\\n\\nWhen Citations is enabled, the API processes user-provided source documents (PDF documents and plain text files) by chunking them into sentences.',\n start_index: 1867,\n end_index: 2030,\n document: 'Citations on the Anthropic API',\n },\n {\n text: \"These chunked sentences, along with user-provided context, are then passed to the model with the user's query. Alternatively, users can provide their own chunks for the source documents.\",\n start_index: 2030,\n end_index: 2218,\n document: 'Citations on the Anthropic API',\n },\n {\n text: 'Claude analyzes the query and generates a response that includes precise citations based on the provided chunks and context for any claims derived from the source material. Cited text will reference source documents to minimize hallucinations.',\n start_index: 2218,\n end_index: 2463,\n document: 'Citations on the Anthropic API',\n },\n {\n text: \"This approach offers superior flexibility and ease of use, as it doesn't require file storage and seamlessly integrates with the Messages API.\",\n start_index: 2463,\n end_index: 2607,\n document: 'Citations on the Anthropic API',\n },\n {\n text: \"Our internal evaluations show that Claude's built-in citation capabilities outperform most custom implementations, increasing recall accuracy by up to 15%.1\\n\\nUse cases\\n\\nWith Citations, developers can create AI solutions that offer enhanced accountability across use cases like:\\n\\n Document summarization: Generate concise summaries of long documents, like case files, with each key point linked back to its original source.\",\n start_index: 1041,\n end_index: 1474,\n document: 'Citations on the Anthropic API',\n },\n {\n text: 'Pricing\\n\\nCitations uses our standard token-based pricing model. While it may use additional input tokens to process documents, users will not pay for output tokens that return the quoted text itself.',\n start_index: 2607,\n end_index: 2811,\n document: 'Citations on the Anthropic API',\n },\n ],\n },\n {\n question: 'How is Citations priced?',\n answer:\n \"According to the document, here are the key points about Citations pricing:\\n\\nCitations follows Anthropic's standard token-based pricing model. While using Citations may require additional input tokens for processing documents, users will not be charged for output tokens that return the quoted text itself.\",\n citations: [\n {\n text: 'Pricing\\n\\nCitations uses our standard token-based pricing model. While it may use additional input tokens to process documents, users will not pay for output tokens that return the quoted text itself.',\n start_index: 2607,\n end_index: 2811,\n document: 'Citations on the Anthropic API',\n },\n ],\n },\n {\n question: 'Which Claude models support Citations?',\n answer:\n \"According to the documents, Citations is available for Claude 3.5 Sonnet and Claude 3.5 Haiku. The feature is generally available on both the Anthropic API and Google Cloud's Vertex AI.\",\n citations: [\n {\n text: 'Get started\\n\\nCitations is now available for the new Claude 3.5 Sonnet and Claude 3.5 Haiku.',\n start_index: 4114,\n end_index: 4209,\n document: 'Citations on the Anthropic API',\n },\n {\n text: 'Citations is generally available on the Anthropic API and Google Cloud\\u2019s Vertex AI.',\n start_index: 315,\n end_index: 400,\n document: 'Citations on the Anthropic API',\n },\n ],\n },\n];\n\nimport DocumentCitations from '@components/prose/DocumentCitations.astro';\nimport sourceMarkdown from './_source.md?raw';\n\n<DocumentCitations qaPairs={qaPairs} sourceMarkdown={sourceMarkdown} />\n\n---\n\nClaude Citations outputs the exact passage from the cited document as well as start and end indices referencing the slices of the original document (as a string) that contain the citations.\n\nWe unpack the inference response from Claude into a data structure for this post like this:\n\nTo make the above UX work as rendered HTML, we need to first add HTML markup to the raw markdown string before rendering, since the cited passages and document indices would no longer match the content after rendering the markdown as HTML.\nTo add the highlighting, we iterate through the citations from end to beginning to avoid interfering with the citation start and end indices.\nWithin each citation, we break the content up by line and wrap each line with <mark> tags, with special case handling for markdown elements like headings and lists.\nFinally, we render that HTML/markdown mix as HTML, resulting in the highlighted document upon selecting a question.\n\nChallenges\n\nFinding a way to add the highlights to the document was the biggest challenge because we must do so before the markdown is rendered as HTML (the citations are in markdown) but we also needed to avoid interfering with the markdown to HTML renderer so that the HTML markup would be added correctly. For example, a common problem I ran into was correct highlighting b",
"canonicalUrl": "https://www.danielcorin.com/posts/2025/document-citations/index"
}