{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreia5mjdvw2oyoddiexohyt7p4l5x3gfay7obsmjmeeaaemy22rsym4",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mpofncv65ne2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreiay7brmow5sydmjdu6cgap7yldhnhr72zkbcbkwxbsxqee27hyusu"
    },
    "mimeType": "image/webp",
    "size": 76812
  },
  "path": "/mongodb/no-messages-table-the-data-model-behind-my-own-claude-based-chatbot-54jp",
  "publishedAt": "2026-07-02T15:47:31.000Z",
  "site": "https://dev.to",
  "tags": [
    "mongodb",
    "ai",
    "claude",
    "tutorial",
    "Néstor Daza",
    "Claudius",
    "Github",
    "prologue",
    "Zod",
    "document embedding",
    "TTL",
    "build this foundation"
  ],
  "textContent": "_This tutorial was written by Néstor Daza._\n\n_This is the second article in a series about building Claudius, my own Claude-based chatbot (Github). The prologue made the case for building it, and for choosing MongoDB as its foundation._\n\nOpen the `conversations` collection in Claudius’ database and you find the usual fields of a thread header but nothing else: a `userId`, a `title`, some `timestamps`, and so on, but no array of messages, no messages collection sitting beside it either! The text of every conversation lives somewhere else entirely, in the LangGraph checkpointer, which I wire up later in this series. This absence is a modeling decision, and how I came up with the database schema for my chatbot is the theme of this article.\n\nIf you come from a relational background, you're used to modeling the data first when designing a database. For a project like this, you would start by finding the entities and normalizing them, and the final schema would come out of the data's structure: a `conversations` table and a `messages` table with a foreign key between them, because that is what the data looks like.\n\nDocument modeling runs the other way. You start from how the application reads and writes, and the shape of the document follows the access patterns. Claudius never reads conversation messages without the agent's full working state wrapped around them, and that state is persisted using the LangGraph checkpointer. A separate `messages` table would add nothing, since the app would always have to join it back to that state on every read. The access pattern says the `messages` belong with the agent state, so that is where they go, and `conversations` are left as the lightweight header the list view actually needs.\n\nThat inversion, modeling around use rather than around the data, runs through everything below.\n\n##  Schema-flexible is not schemaless\n\nThis is the misconception lots of people often carry, and it is worth killing on the way in. A document database does not mean no schema; it means a flexible, use-case-based data model. A relational engine holds the schema for you through column types and constraints; MongoDB will store documents in whatever shape your application gives it, and can validate that shape with a collection validator if you want. Claudius keeps the schema in code, as one Zod schema per collection, because that single definition gives both the runtime check and the TypeScript type.\n\n\n\n    import { z } from \"zod\";\n    import { zObjectId } from \"./common\";\n\n    export const ConversationSchema = z.object({\n      _id: zObjectId.optional(),\n      userId: zObjectId,\n      title: z.string(),\n      modelId: z.string(),\n      createdAt: z.date(),\n      updatedAt: z.date(),\n      archived: z.boolean(),\n      expiresAt: z.date().optional(),\n    });\n\n    export type Conversation = z.infer<typeof ConversationSchema>;\n\n\nThe reason is _drift_. Declare a TypeScript interface for a document and a separate validator for the data coming in, and the two will disagree eventually, while the type system will lie to you with complete confidence. Deriving the type from the validator collapses them into one artifact. One definition gives you the runtime check at the boundary and the compile-time type everywhere else. For a relational reader, this is your DDL (Data Definition Language) and your ORM (Object-Relational Mapping) models together in the same structure, which is something relational stacks rarely pull off.\n\nAnother advantage is that every external input, whether an incoming HTTP body or a model's structured output, is parsed through Zod at the boundary, so by the time data reaches the logic, it is already validated and typed.\n\n##  A quick tour of the model\n\nThere are eight collections in this phase: `users`, `conversations`, `memories`, `documents`, `chunks`, `usage_events`, `settings`, and, arriving later, `jobs`. Two more, `checkpoints` and `checkpoint_writes`, belong entirely to the checkpointer and are never written to directly, so I do not model them. Most of these schemas are unremarkable once you see the access-pattern framing, but two modeling moves are worth pausing on.\n\nThe first is `settings`. It holds a handful of global singletons, each a different shape, each identified by a string `_id`: the member allowlist, the model catalog, the per-role tiers, and the guest spend circuit breaker. They share one collection, so the Zod schema is a discriminated union based on `_id`, and parsing a settings document narrows it to the correct shape from its `_id` literal.\n\n\n\n    export const SettingsSchema = z.discriminatedUnion(\"_id\", [\n      AllowlistSettingsSchema,        // _id: \"allowlist\"\n      ModelCatalogSettingsSchema,     // _id: \"modelCatalog\"\n      TiersSettingsSchema,            // _id: \"tiers\"\n      GuestCircuitBreakerSettingsSchema, // _id: \"guestCircuitBreaker\"\n    ]);\n    export type Settings = z.infer<typeof SettingsSchema>;\n\n\nIn relational terms, this is a single typed configuration table whose rows legitimately carry different columns, which is usually awkward to model relationally without resorting to a JSON column, and here comes out clean and readable.\n\nThe second move is to use document embedding as a data modeling technique. The `users` document carries its `dailyMessageCount` as an embedded object holding the count and its reset time together, rather than as two flat fields or a side table, because the application always reads and writes them together as a unit. That is the document modeling mantra: _data used together is stored together_.\n\n##  Let the database keep the rule\n\nThe most satisfying part of this phase is how little policy I had to write. Three product rules that I would normally enforce in application code are carried by the storage engine instead, because I shaped the data so the engine could hold them.\n\n###  Retention is a TTL index\n\nGuest data in Claudius is ephemeral. Guest-created conversations and memories carry an `expiresAt` date; member and admin documents leave the field off entirely. A TTL (time-to-live) index with `expireAfterSeconds` set to 0 tells MongoDB to delete each document at the exact moment stored in its own `expiresAt` field; documents without this field are never touched.\n\n\n\n    // TTL on the guest-only expiresAt field. expireAfterSeconds:0\n    // means \"expire exactly at the date stored in the field\";\n    // documents without the field are never touched, so member/admin\n    // data is permanent.\n\n    await db\n      .collection(COLLECTIONS.conversations)\n      .createIndex({ expiresAt: 1 }, { name: \"expiresAt_ttl\", expireAfterSeconds: 0 });\n\n    await db\n      .collection(COLLECTIONS.memories)\n      .createIndex({ expiresAt: 1 }, { name: \"expiresAt_ttl\", expireAfterSeconds: 0 });\n\n\nSo, guest ephemerality, a product rule, is enforced by the engine. There is no cron job and no cleanup endpoint to remember to run. The relational equivalent is a scheduled sweep; here, the policy lives in the index.\n\nOne detail ties the TypeScript compiler to the database behavior. With `exactOptionalPropertyTypes` turned on in tsconfig.json, an optional field means the key is absent, not present, and set to `undefined`. The compiler setting and the storage rule reinforce each other.\n\n###  Telemetry is a time-series collection\n\nEvery billable model call writes one `usage_events` document, recording the token counts, the latency, and what the call was for. That data is append-only and gets queried along a few low-cardinality dimensions over a time window, which is the textbook shape for a MongoDB time-series collection. It has a `timeField`, `timestamp`, and a single `metaField` that holds the grouping dimensions (`userId`, `modelId`, `purpose`) nested alongside the measurements.\n\nTwo things follow from this choice. A time-series collection has to be created explicitly with a `timeseries` spec before the first insert (they cannot be created on first write, the way an ordinary collection does), so the index script creates it behind an existence check. Once it exists, the engine's own machinery becomes visible: list the collections and a `system.buckets.usage_events` sits next to `usage_events`, which is where the bucketed storage lives:\n\n\n\n    [\n      {\"name\":\"memories\",\"type\":\"collection\"},\n      {\"name\":\"usage_events\",\"type\":\"timeseries\"},\n      {\"name\":\"system.buckets.usage_events\",\"type\":\"collection\"},\n      {\"name\":\"settings\",\"type\":\"collection\"},\n      {\"name\":\"users\",\"type\":\"collection\"},\n      {\"name\":\"conversations\",\"type\":\"collection\"},\n      {\"name\":\"accounts\",\"type\":\"collection\"},\n      {\"name\":\"chunks\",\"type\":\"collection\"},\n      {\"name\":\"system.views\",\"type\":\"collection\"}\n    ]\n\n\nThe relational analogy is a metrics or rollup table you would otherwise hand-roll and maintain, except the bucketing is automatic and the query you write against it stays simple.\n\n###  A pre-filter field is a security control\n\nThe project's hardest rule is that a vector search pre-filters by `userId` and can never return another user's data. That rule is not bolted onto the search in the application code. It is a property of the index.\n\nBoth `memories` and `chunks` get an Atlas Vector Search index on the `embedding` field, with a `filter` on `userId` (and on `chunks`, a second filter on `documentId`).\n\n\n\n    await ensureVectorIndex(db, COLLECTIONS.memories, \"memories_vector\", [\n      { type: \"vector\", path: \"embedding\", numDimensions: 1024, similarity: \"cosine\" },\n      { type: \"filter\", path: \"userId\" },\n    ], created, skipped);\n\n    await ensureVectorIndex(db, COLLECTIONS.chunks, \"chunks_vector\", [\n      { type: \"vector\", path: \"embedding\", numDimensions: 1024, similarity: \"cosine\" },\n      { type: \"filter\", path: \"userId\" },\n      { type: \"filter\", path: \"documentId\" },\n    ], created, skipped);\n\n\nPre-filtering means the `userId` condition is applied during the indexed search on the same documents being scored, not afterward to whatever the search returns. Post-filtering is not only slower, since you score documents you then throw away, but it is a correctness and security risk, since the wrong documents can slip through ranking and limits before your filter ever runs.\n\nThis is also why `chunks` denormalizes `userId`. A vector search has no notion of a join, so the filter has to sit on the very documents being searched. A chunk belongs to a document that belongs to a user, but the chunk has to carry the owner itself, or the index cannot enforce the boundary. That is the embed-versus-reference tension made concrete: a chunk references its parent document by id and duplicates the one field the security boundary depends on.\n\nTwo gotchas surfaced while creating these indexes, and both are worth saying out loud. A search index can only be built on a collection that already exists, and `chunks` had no classic index, so the collection did not exist yet the first time the script reached for its vector index:\n\n\n\n    MongoServerError: Error retrieving collection UUID and view info ::\n    caused by :: Collection 'claudius.chunks' does not exist. (NamespaceNotFound)\n\n\nThe fix is to add an existence check and create the collection first only if it is missing. The second gotcha is that Atlas builds search indexes asynchronously, so `createSearchIndex` returns when the build is queued, not when it is queryable! We also need a name check that keeps reruns safe while it builds.\n\nThis is the final function to build the indexes from code using the driver's `createSearchIndex`, with the checks that keep the script safe to run on every deploy:\n\n\n\n    async function ensureVectorIndex(\n      db: Db,\n      collectionName: string,\n      indexName: string,\n      fields: Array<Record<string, unknown>>,\n      created: string[],\n      skipped: string[],\n    ): Promise<void> {\n      // A vector search index can only be built on an existing\n      // collection, so create it first if nothing else has (e.g.\n      // chunks has no regular index).\n      const collections = await db\n        .listCollections({ name: collectionName })\n        .toArray();\n      if (collections.length === 0) {\n        await db.createCollection(collectionName);\n      }\n\n      const collection = db.collection(collectionName);\n      const present = await collection.listSearchIndexes().toArray();\n      if (present.some((idx) => idx.name === indexName)) {\n        skipped.push(`${collectionName}.${indexName}`);\n        return;\n      }\n\n      // Atlas builds the index asynchronously; createSearchIndex\n      // returns once the build is queued. The name check above\n      // keeps re-runs idempotent.\n      await collection.createSearchIndex({\n        name: indexName,\n        type: \"vectorSearch\",\n        definition: { fields },\n      });\n      created.push(`${collectionName}.${indexName} (vector)`);\n    }\n\n\n##  The plumbing that keeps the code honest\n\nTwo small things hold the `userId` rule together in practice. First, `userId` is an `ObjectId` everywhere, matching the user’s `_id`, so filters and lookups line up without conversion. Second, no collection is ever queried through a raw driver call. Every access goes through a typed accessor, `conversationsCol()`, `memoriesCol()`, and so on, each returning a typed collection for the right document shape. Centralizing the collection names means a rename happens in one place, and typing each accessor keeps the `userId` filter visible to the compiler at every call site, which is how a rule that lives in prose stays honest in code.\n\nThe whole application and the Auth.js adapter share a single, serverless-safe MongoDB client to a single fixed database name, so there is exactly one connection pool. How that database name quietly caused two separate failures is a good story, and it belongs to the next article, where I stand the foundation up.\n\n##  What this article bought\n\nNo chatbot yet, but the shape of the thing is already decided. Retention, the telemetry layout, and the tenant-isolation boundary are properties of the schema and its indexes now, not code I have to remember to run later. That is the payoff of modeling around the access patterns first: the database carries more of the load because you shaped it to. The data model is the foundational piece of the whole app.\n\nNext, we'll build this foundation up, with identity the client never gets a vote in, and a health check that proves the app can reach Atlas and Bedrock, plus a view on the unglamorous errors that cost me real time. After that comes the streaming chat backbone, which is where the conversation finally gets a place to live, and I open a real checkpoint document to show you the messages that were never in their own collection.",
  "title": "No messages table! The data model behind my own Claude-based chatbot"
}