{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiea3oa3wfyj26aghikgpg5ao5jivfeccskxqawdzxssfh2dnxibte",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mo7ovswm7652"
  },
  "path": "/t/how-can-i-build-a-high-quality-dataset/176571#post_12",
  "publishedAt": "2026-06-14T01:37:27.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "Hugging Face causal language modeling docs",
    "Hugging Face LLM course: training a causal language model",
    "Fewer Truncations Improve Language Modeling",
    "In-context Pretraining",
    "Rethinking KenLM: Good and Bad Model Ensembles for Efficient Text Quality Filtering in Large Web Corpora"
  ],
  "textContent": "Hmm, probably:\n\n* * *\n\n## Short answer\n\nFor the chunking question:\n\n> Orphaned references are not catastrophic if they are rare, but they are not ideal high-quality CPT data.\n\nIf a chunk starts with something like `او`, `این`, `آن`, `همین‌جا`, etc., and the referent is outside the chunk, the model cannot really learn the reference properly from that sample. It just sees an incomplete continuation.\n\nFor the Good/Bad n-gram question:\n\n> Good and Bad should not be defined only as “surface-clean vs surface-dirty.” They should be defined by imitation target.\n\nIn other words:\n\n\n    Good data = text you want the model to imitate.\n    Bad data = text you want the filter to detect and remove.\n\n\nSo yes, after surface cleaning, the distinction becomes more about:\n\n  * semantic completeness\n  * grammatical correctness\n  * natural Persian\n  * self-containedness\n  * target register\n  * whether the text is useful for your model’s target behavior\n\n\n\n* * *\n\n## 1. About orphaned references in chunks\n\nThe example:\n\n\n    از همین‌جا بود که تاریخ ویکی شروع شد\n\n\nis not necessarily bad Persian. But it is context-dependent.\n\nThe phrase `همین‌جا` points to something before it. If the previous context is missing, then the chunk is incomplete.\n\nSimilarly:\n\n\n    او سپس به تهران رفت.\n\n\nis grammatical, but if the dataset chunk does not mention who `او` is, the model receives an incomplete training sample.\n\nFor CPT, this matters because causal language modeling is next-token prediction. The model learns from the previous tokens in the sequence. If the needed left context is missing, the model cannot infer the reference correctly from that sample.\n\nRelated docs:\n\n  * Hugging Face causal language modeling docs\n  * Hugging Face LLM course: training a causal language model\n\n\n\nSo I would classify these chunks as:\n\nChunk type | Use for Good CPT?\n---|---\ncomplete paragraph | yes\ncomplete sentence with enough context | yes\nsentence starting with unresolved pronoun | maybe, but lower quality\nshort fragment with missing referent | usually reject or merge\nreference/citation tail | reject or use as Bad data\ntitle/header/list fragment | maybe separate category, not normal prose\n\n* * *\n\n## 2. This is not a “pronoun problem” only\n\nThe issue is broader than pronouns.\n\nIt is about whether the chunk is **self-contained enough**.\n\nProblematic context-dependent starts include:\n\n\n    او ...\n    آن‌ها ...\n    این موضوع ...\n    در نتیجه ...\n    از همین‌جا ...\n    به همین دلیل ...\n    همان‌طور که گفته شد ...\n    این مسئله ...\n    چنین رویدادی ...\n    وی ...\n    نامبرده ...\n\n\nThese are normal Persian expressions, but they often depend on previous context.\n\nThey are not automatically bad. But if the antecedent or explanation is outside the chunk, the sample is weaker.\n\nA useful rule:\n\n> A Good CPT chunk does not need to be globally complete, but it should be locally coherent.\n\n* * *\n\n## 3. Prefer document-aware chunking over blind fixed-size chunking\n\nI would not start by chopping everything into fixed token lengths.\n\nBetter:\n\n\n    article\n      -> section\n      -> paragraph\n      -> sentence\n      -> clean units\n      -> pack units into training sequences\n\n\nThen you can pack clean units up to your target sequence length.\n\nThis is different from:\n\n\n    raw text\n      -> every N tokens\n      -> chunk\n\n\nBlind fixed-length chunking often creates:\n\n  * broken sentence starts\n  * broken endings\n  * orphaned pronouns\n  * cut-off lists\n  * reference tails\n  * citation garbage\n  * chunks that start mid-thought\n\n\n\nResearch on packing/truncation also points in this direction. For example, Fewer Truncations Improve Language Modeling argues that naive concatenation and fixed-length splitting can damage document integrity and that length-aware packing can reduce unnecessary truncation.\n\nAlso, In-context Pretraining discusses the weakness of randomly concatenating short documents and shows that related/coherent document ordering can improve tasks requiring contextual reasoning.\n\nYou do not need to implement those papers exactly, but the practical lesson is useful:\n\n> Preserve coherent units when possible.\n\n* * *\n\n## 4. What to do with short incomplete chunks\n\nFor each short chunk, choose one of four actions.\n\nCase | Action\n---|---\nshort but complete | keep\nstarts with unresolved pronoun | merge with previous paragraph/sentence\ngood sentence but missing previous context | keep only if not too frequent, or lower priority\nfragment/citation/list tail | reject\ngood prose + bad tail | truncate tail\ntoo short to be meaningful | reject\n\nExample:\n\n\n    او سپس به تهران رفت.\n\n\nIf the previous sentence is:\n\n\n    علی در سال ۱۳۵۷ از اصفهان خارج شد.\n\n\nthen merge:\n\n\n    علی در سال ۱۳۵۷ از اصفهان خارج شد. او سپس به تهران رفت.\n\n\nThat is much better than training on the orphaned sentence alone.\n\nAnother example:\n\n\n    از همین‌جا بود که تاریخ ویکی شروع شد.\n\n\nIf the previous sentence identifies the place/context, merge it.\n\nIf not, reject or mark it as lower-quality.\n\n* * *\n\n## 5. Use EOS or document separators\n\nIf you pack multiple documents into one training sequence, add a clear separator.\n\nUsually that means EOS.\n\nConceptually:\n\n\n    document A text <eos> document B text <eos> document C text <eos>\n\n\nThis is better than making the model think unrelated documents are one continuous article.\n\nIf the tokenizer/model has an EOS token, use it consistently.\n\nIf you are doing CPT with Qwen, follow the tokenizer/model convention rather than inventing a random separator.\n\nThe goal is:\n\n\n    The model should know where one document ends.\n\n\nnot:\n\n\n    The model should continue one unrelated article as if it were the same text.\n\n\n* * *\n\n## 6. Packing vs padding\n\nThere is a trade-off.\n\nMethod | Benefit | Risk\n---|---|---\npure padding | preserves document boundaries | inefficient\nnaive concatenation | efficient | can mix unrelated contexts\nfixed token chunks | simple | breaks documents/sentences\nparagraph-aware packing | good balance | more preprocessing\nlength-aware packing | better integrity | more implementation work\n\nFor your case, I would start simple:\n\n\n    paragraph-aware packing + EOS between documents\n\n\nNot necessarily complicated optimization.\n\nA practical algorithm:\n\n\n    1. split article into paragraphs\n    2. clean each paragraph\n    3. reject bad or too-short paragraphs\n    4. merge neighboring small paragraphs from the same article\n    5. add EOS at article boundary\n    6. pack clean units until max_length\n    7. avoid cutting inside sentence if possible\n\n\n* * *\n\n## 7. How strict should you be?\n\nDo not over-clean to the point that you throw away most natural text.\n\nBut for the Good corpus, be conservative.\n\nSuggested rule:\n\n\n    If uncertain, do not put it in Good.\n\n\nYou can put uncertain data into:\n\n\n    maybe/\n\n\nand inspect later.\n\nPossible buckets:\n\n\n    good/\n    bad/\n    maybe/\n    too_short/\n    reference_tail/\n    needs_previous_context/\n\n\nThis makes the pipeline easier to debug.\n\n* * *\n\n## 8. Defining Good and Bad data\n\nFor the Good/Bad n-gram setup, I would define the labels by imitation target.\n\n### Good data\n\nGood data is text that satisfies most of these:\n\n  * natural Iranian Persian\n  * grammatically correct\n  * semantically coherent\n  * complete enough locally\n  * clear sentence boundaries\n  * clear paragraph boundaries\n  * not citation/reference garbage\n  * not boilerplate\n  * not duplicated\n  * not overly archaic or rare if outside target\n  * not too fragmented\n  * target register matches your goal\n  * something you would be happy for the model to imitate\n\n\n\n### Bad data\n\nBad data is text that matches patterns you want to filter:\n\n  * reference/citation tails\n  * orphan fragments\n  * broken chunks\n  * malformed number sequences\n  * mixed-language bibliography\n  * OCR-like text\n  * repeated boilerplate\n  * random lists/tables\n  * source titles glued into prose\n  * incomplete sentences\n  * nonsense or semantically broken text\n  * grammatically unnatural text\n  * wrong register for your target\n  * archaic/rare language if you intentionally exclude it\n\n\n\nSo yes, after surface cleaning, the distinction becomes more semantic and linguistic.\n\nBut it is still not only “semantic quality.” It is also:\n\n\n    Would I want the final model to produce text like this?\n\n\n* * *\n\n## 9. Good LM / Bad LM\n\nThe Good/Bad KenLM idea is:\n\n\n    Good LM learns the pattern of text you want.\n    Bad LM learns the pattern of text you do not want.\n\n\nUseful reference:\n\n  * Rethinking KenLM: Good and Bad Model Ensembles for Efficient Text Quality Filtering in Large Web Corpora\n\n\n\nA simple setup:\n\nModel | Train on\n---|---\nGood KenLM | clean Persian paragraphs\nBad KenLM | rejected noisy/fractured/reference-like chunks\n\nThen for a new candidate:\n\n\n    Good LM likes it\n    Bad LM dislikes it\n    => probably good\n\n\nIf both like it, inspect.\n\nIf Good LM dislikes it and Bad LM likes it, reject.\n\nIf both dislike it, it may be out-of-domain or unusual.\n\n* * *\n\n## 10. Do not make Good too narrow\n\nOne warning: if Good data is only formal Wikipedia prose, your Good LM will prefer Wikipedia-like Persian.\n\nThat may be okay for CPT, but your final assistant may need:\n\n  * educational Persian\n  * student-friendly Persian\n  * conversational Persian\n  * grammar-tutor Persian\n  * technical Persian\n  * simple explanation style\n\n\n\nSo you may want several Good categories:\n\n\n    good_formal/\n    good_educational/\n    good_conversational/\n    good_technical/\n\n\nOr at least make sure Good is not only one style.\n\nOtherwise your filter may reject useful informal/student-like Persian just because it is not Wikipedia-like.\n\n* * *\n\n## 11. Do not make Bad too broad\n\nBad should not mean “anything different from Wikipedia.”\n\nBad should mean:\n\n\n    text I do not want the model to imitate\n\n\nExamples:\n\n\n    bad_reference_tail\n    bad_broken_number\n    bad_ocr\n    bad_mixed_language_garbage\n    bad_boilerplate\n    bad_contextless_fragment\n    bad_duplicate\n\n\nDo not put valid conversational Persian into Bad just because it has a different style.\n\nOtherwise the filter may remove exactly the kind of natural user-language you later want.\n\n* * *\n\n## 12. Practical labeling guide\n\nI would use labels like this.\n\nLabel | Meaning | Use\n---|---|---\n`good_formal` | clean formal prose | Good LM\n`good_educational` | clean explanation/tutorial prose | Good LM\n`good_conversational` | natural dialogue-like Persian | maybe Good LM or separate LM\n`bad_reference_tail` | bibliography/citation garbage | Bad LM\n`bad_fragment` | incomplete contextless chunk | Bad LM\n`bad_boilerplate` | repeated non-content | Bad LM\n`bad_mixed_language` | accidental mixed-language garbage | Bad LM\n`bad_number_noise` | malformed number/date fragments | Bad LM\n`maybe` | uncertain | manual review\n\nThis is better than just:\n\n\n    good\n    bad\n\n\nbecause later you can inspect which kind of bad data is leaking through.\n\n* * *\n\n## 13. Specific rule for orphaned pronouns\n\nFor chunks with pronouns or context markers:\n\n\n    او\n    وی\n    آن‌ها\n    این\n    آن\n    همین\n    چنین\n    نامبرده\n\n\nI would use a simple rule:\n\n\n    If the chunk is short and starts with one of these, try to merge with previous sentence/paragraph.\n    If no previous context exists, mark as context-dependent.\n    If context-dependent and short, reject from Good.\n\n\nNot every pronoun is bad. For example:\n\n\n    علی دانش‌آموز کلاس هفتم است. او به ریاضی علاقه دارد.\n\n\nThis is good.\n\nBut:\n\n\n    او به ریاضی علاقه دارد.\n\n\nalone is weak.\n\n* * *\n\n## 14. Example decision table\n\nExample | Decision\n---|---\n`علی دانش‌آموز کلاس هفتم است. او به ریاضی علاقه دارد.` | Good\n`او به ریاضی علاقه دارد.` | Maybe/reject unless merged\n`از همین‌جا بود که تاریخ ویکی شروع شد.` | Maybe; needs previous context\n`۲ ۸ مرداد ۱ ۳ ۳ ۲ BBC Persian Modern Iran p۱ ۲ ۲` | Bad\n`منابع طبیعی ایران بسیار متنوع است.` | Good\n`منابع: BBC Persian, Abrahamian, Modern Iran...` | Bad reference tail\n`این موضوع در بخش قبل توضیح داده شد.` | Context-dependent; maybe reject\n`در این درس، مفهوم فاعل را با مثال توضیح می‌دهیم.` | Good educational\n\n* * *\n\n## 15. A practical pipeline\n\nI would do something like:\n\n\n    raw article\n      -> normalize\n      -> split into sections\n      -> split into paragraphs\n      -> split into sentences\n      -> remove reference sections\n      -> remove/truncate citation tails\n      -> detect context-dependent starts\n      -> merge with previous paragraph if needed\n      -> reject very short orphan fragments\n      -> assign good/bad/maybe labels\n      -> train Good/Bad n-gram models\n\n\nFor chunk construction:\n\n\n    clean paragraphs\n      -> merge small adjacent paragraphs from same article\n      -> add EOS at article boundary\n      -> pack to max_length\n      -> avoid cutting inside sentence when possible\n\n\n* * *\n\n## 16. Suggested thresholds\n\nThese are only starting points.\n\nRule | Possible starting point\n---|---\ntoo short | fewer than 20-30 Persian tokens\ntoo digit-heavy | digit ratio above 15-20%\ntoo Latin-heavy | Latin ratio above 20-30%, unless technical text\norphan start | starts with pronoun/context marker and very short\nlow Persian ratio | reject unless intended mixed text\ntoo repetitive | repeated line or repeated n-gram\nsuspicious tail | truncate after reference-like pattern\n\nDo not trust the thresholds blindly.\n\nSample accepted/rejected data and adjust.\n\n* * *\n\n## 17. How much orphaned data is harmful?\n\nIf rare, probably not a disaster.\n\nLarge corpora always contain some imperfect examples.\n\nBut if many chunks are orphaned fragments, the model may learn:\n\n  * incomplete starts\n  * weak local coherence\n  * continuation from missing context\n  * strange discourse transitions\n  * bad paragraph openings\n\n\n\nFor a small 0.8B model, I would be stricter than for a huge model.\n\nSmall models have less capacity to average away noise.\n\nSo:\n\n\n    For Good CPT corpus:\n      be conservative.\n\n    For broad noisy corpus:\n      maybe keep some.\n\n    For final high-quality Persian corpus:\n      reject or merge orphan fragments.\n\n\n* * *\n\n## 18. Bottom line\n\nFor /10:\n\n> Orphaned pronouns and context-dependent chunks are not catastrophic if rare, but they are not high-quality CPT examples. Prefer article/section/paragraph-aware chunking, merge short context-dependent chunks with previous context, use EOS between documents, and reject very short orphan fragments.\n\nFor /11:\n\n> Good/Bad data should be defined by imitation target, not just surface cleanliness. Good = text you want the model to imitate. Bad = text you want your filter to detect and remove.\n\nThe simplest rule is:\n\n\n    Good data should be locally coherent Persian prose.\n    Bad data should represent the patterns you want to eliminate.\n",
  "title": "How can i build a High Quality dataset?"
}