{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreidgpbwegjd5iyyt6jh35sxiwoo4xx5vj3d3nzoly5bhy4cjcgouru",
    "uri": "at://did:plc:lk3jfj3zq4k4wxnk474axylu/app.bsky.feed.post/3mhsqy3jy3wr2"
  },
  "path": "/t/cosine-similarity-variance-when-migrating-from-text-embedding-ada-002-to-cosine-similarity-variance-when-migrating-from-text-embedding-ada-002-to-text-embedding-3-small/1377685#post_1",
  "publishedAt": "2026-03-24T14:04:45.000Z",
  "site": "https://community.openai.com",
  "textContent": "We have a tutoring chatbot that relies on **embedding-based relevance scoring** for user queries. We are in the process of evaluating a migration from **`text-embedding-ada-002`** to **`text-embedding-3-small`**. Although changes in cosine similarity values across embedding models are expected, our evaluation indicates that similarity scores produced by `text-embedding-3-small` are **significantly lower** and **not consistently ordered** relative to those from `text-embedding-ada-002`.\n\n## Issue Summary\n\nFor the same query–context pairs, we observed **significant and inconsistent differences in cosine similarity scores** between the legacy embedding model **`text-embedding-ada-002`** and the newer model **`text-embedding-3-small`**.\n\nIn several cases, cosine similarity values produced by `text-embedding-3-small` are **substantially lower** than those produced by `text-embedding-ada-002`, and the **relative ordering of similarity scores across queries is not consistent** between the two models.\n\nThis behavior raises concerns that **semantic relevance scoring may be altered** when migrating from `ada-002` to `text-embedding-3-small`.\n\n## Issue Details (With Example)\n\n### Context\n\n\n    Question shown to the student:\n    <p>Find the prime factorization of the following number.</p> <p>(15)</p>\n\n    Solution of the question is:\n    <p>Factor (15) into two factors, (3) and (5).</p>\n\n\n### Queries Evaluated\n\n  1. **Query 1:**\n“The best statistical software to tackle this problem would be…”\n  2. **Query 2:**\n“How does this concept apply to everyday situations?”\n  3. **Query 3:**\n“How does this topic connect to other areas of statistics or mathematics?”\n\n\n\n## Cosine Similarity Results\n\n### text-embedding-ada-002\n\nQuery | Cosine Similarity\n---|---\nQuery 1 | 0.774218917944234\nQuery 2 | 0.781920253363479\nQuery 3 | 0.789893634044595\n\n**Observation:**\nCosine similarity values show a **clear increasing trend** across the three queries.\n\n### text-embedding-3-small\n\nQuery | Cosine Similarity\n---|---\nQuery 1 | 0.247923658700569\nQuery 2 | 0.195844709264796\nQuery 3 | 0.217488219437886\n\n**Observation:**\nCosine similarity values are **much lower overall** and do **NOT follow a consistent increasing or decreasing order** across the same queries.\n\n## Key Observations\n\n  * The absolute cosine similarity scores from `text-embedding-3-small` are **significantly lower** than those from `text-embedding-ada-002` for the same query–context pairs.\n  * The **relative ranking of queries by similarity differs between the two models**.\n  * In `ada-002`, similarity scores increase monotonically across the example queries.\n  * In `text-embedding-3-small`, similarity scores fluctuate (increase and decrease), even when the same trend is expected.\n  * This inconsistency suggests that **semantic relevance interpretation differs substantially** between the old and new models.\n\n\n\n## Conclusion / Concern\n\nFor applications relying on cosine similarity thresholds, ranking, or relevance ordering, this change may lead to **unexpected or degraded results** after migration.\n\nClarification is requested on whether:\n\n  * There are **recommended normalization, threshold, or evaluation adjustments** when switching to the new embedding models.\n*Given that our current cosine similarity threshold with the legacy embedding model **`text-embedding-ada-002`** is **0.7** , is it appropriate to use a threshold of **0.2** after upgrading to **`text-embedding-3-small`** , or is a different threshold recommended?\n\n",
  "title": "Cosine Similarity Variance When Migrating from text-embedding-ada-002 to Cosine Similarity Variance When Migrating from text-embedding-ada-002 to text-embedding-3-small"
}