{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreif7l77ds7gonclee2y7p77gvhsxzqpavkcf4kovlclamhet5izgqm",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mobyqgowk4b2"
  },
  "path": "/t/how-can-i-build-a-high-quality-dataset/176571#post_14",
  "publishedAt": "2026-06-14T22:32:31.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "Jurafsky and Martin’s n-gram chapter",
    "Rethinking KenLM"
  ],
  "textContent": "Hmm, probably:\n\n* * *\n\nNo, I would not say that n-gram models automatically perform better just because the input text is shorter.\n\nA 5-gram model is local in a different sense: it mainly uses the previous 4 tokens/words when estimating the next one. So it can score a long article, but the score is still built from many local n-gram probabilities.\n\nAlso, perplexity is normally length-normalized. For example, Jurafsky and Martin’s n-gram chapter describes perplexity as a normalized version of the probability of the test set. So a long text is not automatically invalid just because it is long.\n\nThe real issue is not:\n\n\n    long text is bad for n-gram models\n\n\nThe real issue is:\n\n\n    whole-document averaging can hide local garbage\n\n\nFor example, if an article is:\n\n\n    good paragraph\n    good paragraph\n    good paragraph\n    bad reference tail\n\n\nthen the whole-article average perplexity may still look acceptable because the good paragraphs dominate the score.\n\nSo I would separate **training** and **filtering** :\n\nStep | Unit I would use\n---|---\nTrain Good KenLM | clean paragraphs or clean articles are both okay\nTrain Bad KenLM | bad tails, broken references, noisy chunks\nScore/filter candidates | paragraph-level or sliding-window level\nDetect local garbage | sentence/window level\nFinal CPT packing | merge clean paragraphs and add EOS/document separators\n\nSo yes: if the long articles are already clean, training KenLM on them is fine.\n\nBut for filtering, I would avoid relying only on a single whole-article score. I would score smaller units too:\n\n\n    article score\n    paragraph scores\n    sliding-window scores\n\n\nA practical rule:\n\n\n    Use long clean text for training.\n    Use local scores for filtering.\n\n\nThis also fits the Good/Bad KenLM idea. The Rethinking KenLM paper uses a Good KenLM trained on high-quality data and a Bad KenLM trained on low-quality data. That does not require all input texts to be short; the important part is that the Good model learns the pattern of text you want, and the Bad model learns the pattern of text you want to remove.\n\nSo my short version would be:\n\n> Long clean text is fine for training KenLM. Whole-document averaging is the danger. For filtering Persian CPT data, use paragraph-level or sliding-window scoring so local reference garbage does not get hidden by good surrounding text.",
  "title": "How can i build a High Quality dataset?"
}