{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreigkdqml4b2sjlcahcu4t34rtwqsgs55lzqa5upgv42wljpvkglbke",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mo6ndwr7jmn2"
  },
  "path": "/t/how-can-i-build-a-high-quality-dataset/176571#post_10",
  "publishedAt": "2026-06-13T16:04:01.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "Hello!\n\nI have a question regarding CPT dataset preparation. I recently refined my dataset by removing rare or archaic languages, such as Ancient Persian, because I was concerned that the tokenizer wouldn’t handle them efficiently, which might negatively impact the model’s learning. I also normalized number formats and other technical details.\n\nFinally, I chunked the large texts into smaller segments. My general rule was to chunk everything unless it was a specific structural element, like a rule or an email address. However, I’ve run into a context issue.\n\nFor example, in a long article about Wikipedia, one of the resulting chunks is: _\"از همین‌جا بود که تاریخ ویکی شروع شد_ \". If I feed these disconnected fragments into the model, how do I handle incomplete or discontinued text? Specifically, if a short text in the dataset uses a pronoun like _“او”,_ but the article is cut off and doesn’t mention who _“او”_ refers to, how does the model learn to resolve that reference? Does having these “orphaned” references in the dataset harm the training process?",
  "title": "How can i build a High Quality dataset?"
}