{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreifpkl7uvksvjetlnjpkvecdpzy75isewiqiy5bpvymnzl2vp23v7a",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mfvdkegtpn72"
  },
  "path": "/t/catastrophic-forgetting-of-language-models/173860#post_1",
  "publishedAt": "2026-02-27T17:45:47.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "To all the awesome experts in AI/ML out there. i need a favor.\nI realized there is a gap in Language Models (SLMs/LLMs) remembering the data continuously which is termed as ‘catastrophic forgetting’.\n\nTo solve that problem I came up with an adapter called Constrained Residual Mixing Adapter (CRMA) that enables continual learning. I tested it on Tiny Llama 1.1B and Mistral 7B — the result: -0.1% drift across 4 sequential\ndomains. Essentially zero forgetting.\n\nCRMA: -0.1% drift. Naive: +351% forgetting. Same model, same data, same hardware.\n\nHolds at both 1.1B and 7B. No replay, no EWC, no KD needed.\n● CRMA Modular vs Naive — Mistral 7B (4 sequential domains)\n\n┌───────── ┬────────────┬──\n│ Task │ CRMA Drift │ Naive Forgetting │\n├───────── ┼────────────┼──\n│ Medical │ -0.2% │ +228% │\n├───────── ┼────────────┤\n│ Legal │ -0.1% │ +593% │\n├───────── ┼────────────\n│ Code │ -0.1% │ +233% │\n├───────── ────────────\n│ Finance │ +0.0% — │\n├─────────┼─────────────┤\n│ Average │ -0.1% │ +351% │\n└─────────┴────────────┘\n\nNow the favor - If you’re interested in independently verifying these results, I’d love to hear from you. DM me and I’ll share what you need to reproduce it. Thank you. and best wishes",
  "title": "Catastrophic Forgetting of language models"
}