How can i build a High Quality dataset?

Hello!

I have a question regarding CPT dataset preparation. I recently refined my dataset by removing rare or archaic languages, such as Ancient Persian, because I was concerned that the tokenizer wouldn’t handle them efficiently, which might negatively impact the model’s learning. I also normalized number formats and other technical details.

Finally, I chunked the large texts into smaller segments. My general rule was to chunk everything unless it was a specific structural element, like a rule or an email address. However, I’ve run into a context issue.

For example, in a long article about Wikipedia, one of the resulting chunks is: "از همین‌جا بود که تاریخ ویکی شروع شد ". If I feed these disconnected fragments into the model, how do I handle incomplete or discontinued text? Specifically, if a short text in the dataset uses a pronoun like “او”, but the article is cut off and doesn’t mention who “او” refers to, how does the model learn to resolve that reference? Does having these “orphaned” references in the dataset harm the training process?