How can i build a High Quality dataset?
Hmm, probably:
Short answer
For the chunking question:
Orphaned references are not catastrophic if they are rare, but they are not ideal high-quality CPT data.
If a chunk starts with something like او, این, آن, همینجا, etc., and the referent is outside the chunk, the model cannot really learn the reference properly from that sample. It just sees an incomplete continuation.
For the Good/Bad n-gram question:
Good and Bad should not be defined only as “surface-clean vs surface-dirty.” They should be defined by imitation target.
In other words:
Good data = text you want the model to imitate.
Bad data = text you want the filter to detect and remove.
So yes, after surface cleaning, the distinction becomes more about:
- semantic completeness
- grammatical correctness
- natural Persian
- self-containedness
- target register
- whether the text is useful for your model’s target behavior
1. About orphaned references in chunks
The example:
از همینجا بود که تاریخ ویکی شروع شد
is not necessarily bad Persian. But it is context-dependent.
The phrase همینجا points to something before it. If the previous context is missing, then the chunk is incomplete.
Similarly:
او سپس به تهران رفت.
is grammatical, but if the dataset chunk does not mention who او is, the model receives an incomplete training sample.
For CPT, this matters because causal language modeling is next-token prediction. The model learns from the previous tokens in the sequence. If the needed left context is missing, the model cannot infer the reference correctly from that sample.
Related docs:
- Hugging Face causal language modeling docs
- Hugging Face LLM course: training a causal language model
So I would classify these chunks as:
| Chunk type | Use for Good CPT? |
|---|---|
| complete paragraph | yes |
| complete sentence with enough context | yes |
| sentence starting with unresolved pronoun | maybe, but lower quality |
| short fragment with missing referent | usually reject or merge |
| reference/citation tail | reject or use as Bad data |
| title/header/list fragment | maybe separate category, not normal prose |
2. This is not a “pronoun problem” only
The issue is broader than pronouns.
It is about whether the chunk is self-contained enough.
Problematic context-dependent starts include:
او ...
آنها ...
این موضوع ...
در نتیجه ...
از همینجا ...
به همین دلیل ...
همانطور که گفته شد ...
این مسئله ...
چنین رویدادی ...
وی ...
نامبرده ...
These are normal Persian expressions, but they often depend on previous context.
They are not automatically bad. But if the antecedent or explanation is outside the chunk, the sample is weaker.
A useful rule:
A Good CPT chunk does not need to be globally complete, but it should be locally coherent.
3. Prefer document-aware chunking over blind fixed-size chunking
I would not start by chopping everything into fixed token lengths.
Better:
article
-> section
-> paragraph
-> sentence
-> clean units
-> pack units into training sequences
Then you can pack clean units up to your target sequence length.
This is different from:
raw text
-> every N tokens
-> chunk
Blind fixed-length chunking often creates:
- broken sentence starts
- broken endings
- orphaned pronouns
- cut-off lists
- reference tails
- citation garbage
- chunks that start mid-thought
Research on packing/truncation also points in this direction. For example, Fewer Truncations Improve Language Modeling argues that naive concatenation and fixed-length splitting can damage document integrity and that length-aware packing can reduce unnecessary truncation.
Also, In-context Pretraining discusses the weakness of randomly concatenating short documents and shows that related/coherent document ordering can improve tasks requiring contextual reasoning.
You do not need to implement those papers exactly, but the practical lesson is useful:
Preserve coherent units when possible.
4. What to do with short incomplete chunks
For each short chunk, choose one of four actions.
| Case | Action |
|---|---|
| short but complete | keep |
| starts with unresolved pronoun | merge with previous paragraph/sentence |
| good sentence but missing previous context | keep only if not too frequent, or lower priority |
| fragment/citation/list tail | reject |
| good prose + bad tail | truncate tail |
| too short to be meaningful | reject |
Example:
او سپس به تهران رفت.
If the previous sentence is:
علی در سال ۱۳۵۷ از اصفهان خارج شد.
then merge:
علی در سال ۱۳۵۷ از اصفهان خارج شد. او سپس به تهران رفت.
That is much better than training on the orphaned sentence alone.
Another example:
از همینجا بود که تاریخ ویکی شروع شد.
If the previous sentence identifies the place/context, merge it.
If not, reject or mark it as lower-quality.
5. Use EOS or document separators
If you pack multiple documents into one training sequence, add a clear separator.
Usually that means EOS.
Conceptually:
document A text <eos> document B text <eos> document C text <eos>
This is better than making the model think unrelated documents are one continuous article.
If the tokenizer/model has an EOS token, use it consistently.
If you are doing CPT with Qwen, follow the tokenizer/model convention rather than inventing a random separator.
The goal is:
The model should know where one document ends.
not:
The model should continue one unrelated article as if it were the same text.
6. Packing vs padding
There is a trade-off.
| Method | Benefit | Risk |
|---|---|---|
| pure padding | preserves document boundaries | inefficient |
| naive concatenation | efficient | can mix unrelated contexts |
| fixed token chunks | simple | breaks documents/sentences |
| paragraph-aware packing | good balance | more preprocessing |
| length-aware packing | better integrity | more implementation work |
For your case, I would start simple:
paragraph-aware packing + EOS between documents
Not necessarily complicated optimization.
A practical algorithm:
1. split article into paragraphs
2. clean each paragraph
3. reject bad or too-short paragraphs
4. merge neighboring small paragraphs from the same article
5. add EOS at article boundary
6. pack clean units until max_length
7. avoid cutting inside sentence if possible
7. How strict should you be?
Do not over-clean to the point that you throw away most natural text.
But for the Good corpus, be conservative.
Suggested rule:
If uncertain, do not put it in Good.
You can put uncertain data into:
maybe/
and inspect later.
Possible buckets:
good/
bad/
maybe/
too_short/
reference_tail/
needs_previous_context/
This makes the pipeline easier to debug.
8. Defining Good and Bad data
For the Good/Bad n-gram setup, I would define the labels by imitation target.
Good data
Good data is text that satisfies most of these:
- natural Iranian Persian
- grammatically correct
- semantically coherent
- complete enough locally
- clear sentence boundaries
- clear paragraph boundaries
- not citation/reference garbage
- not boilerplate
- not duplicated
- not overly archaic or rare if outside target
- not too fragmented
- target register matches your goal
- something you would be happy for the model to imitate
Bad data
Bad data is text that matches patterns you want to filter:
- reference/citation tails
- orphan fragments
- broken chunks
- malformed number sequences
- mixed-language bibliography
- OCR-like text
- repeated boilerplate
- random lists/tables
- source titles glued into prose
- incomplete sentences
- nonsense or semantically broken text
- grammatically unnatural text
- wrong register for your target
- archaic/rare language if you intentionally exclude it
So yes, after surface cleaning, the distinction becomes more semantic and linguistic.
But it is still not only “semantic quality.” It is also:
Would I want the final model to produce text like this?
9. Good LM / Bad LM
The Good/Bad KenLM idea is:
Good LM learns the pattern of text you want.
Bad LM learns the pattern of text you do not want.
Useful reference:
- Rethinking KenLM: Good and Bad Model Ensembles for Efficient Text Quality Filtering in Large Web Corpora
A simple setup:
| Model | Train on |
|---|---|
| Good KenLM | clean Persian paragraphs |
| Bad KenLM | rejected noisy/fractured/reference-like chunks |
Then for a new candidate:
Good LM likes it
Bad LM dislikes it
=> probably good
If both like it, inspect.
If Good LM dislikes it and Bad LM likes it, reject.
If both dislike it, it may be out-of-domain or unusual.
10. Do not make Good too narrow
One warning: if Good data is only formal Wikipedia prose, your Good LM will prefer Wikipedia-like Persian.
That may be okay for CPT, but your final assistant may need:
- educational Persian
- student-friendly Persian
- conversational Persian
- grammar-tutor Persian
- technical Persian
- simple explanation style
So you may want several Good categories:
good_formal/
good_educational/
good_conversational/
good_technical/
Or at least make sure Good is not only one style.
Otherwise your filter may reject useful informal/student-like Persian just because it is not Wikipedia-like.
11. Do not make Bad too broad
Bad should not mean “anything different from Wikipedia.”
Bad should mean:
text I do not want the model to imitate
Examples:
bad_reference_tail
bad_broken_number
bad_ocr
bad_mixed_language_garbage
bad_boilerplate
bad_contextless_fragment
bad_duplicate
Do not put valid conversational Persian into Bad just because it has a different style.
Otherwise the filter may remove exactly the kind of natural user-language you later want.
12. Practical labeling guide
I would use labels like this.
| Label | Meaning | Use |
|---|---|---|
good_formal |
clean formal prose | Good LM |
good_educational |
clean explanation/tutorial prose | Good LM |
good_conversational |
natural dialogue-like Persian | maybe Good LM or separate LM |
bad_reference_tail |
bibliography/citation garbage | Bad LM |
bad_fragment |
incomplete contextless chunk | Bad LM |
bad_boilerplate |
repeated non-content | Bad LM |
bad_mixed_language |
accidental mixed-language garbage | Bad LM |
bad_number_noise |
malformed number/date fragments | Bad LM |
maybe |
uncertain | manual review |
This is better than just:
good
bad
because later you can inspect which kind of bad data is leaking through.
13. Specific rule for orphaned pronouns
For chunks with pronouns or context markers:
او
وی
آنها
این
آن
همین
چنین
نامبرده
I would use a simple rule:
If the chunk is short and starts with one of these, try to merge with previous sentence/paragraph.
If no previous context exists, mark as context-dependent.
If context-dependent and short, reject from Good.
Not every pronoun is bad. For example:
علی دانشآموز کلاس هفتم است. او به ریاضی علاقه دارد.
This is good.
But:
او به ریاضی علاقه دارد.
alone is weak.
14. Example decision table
| Example | Decision |
|---|---|
علی دانشآموز کلاس هفتم است. او به ریاضی علاقه دارد. |
Good |
او به ریاضی علاقه دارد. |
Maybe/reject unless merged |
از همینجا بود که تاریخ ویکی شروع شد. |
Maybe; needs previous context |
۲ ۸ مرداد ۱ ۳ ۳ ۲ BBC Persian Modern Iran p۱ ۲ ۲ |
Bad |
منابع طبیعی ایران بسیار متنوع است. |
Good |
منابع: BBC Persian, Abrahamian, Modern Iran... |
Bad reference tail |
این موضوع در بخش قبل توضیح داده شد. |
Context-dependent; maybe reject |
در این درس، مفهوم فاعل را با مثال توضیح میدهیم. |
Good educational |
15. A practical pipeline
I would do something like:
raw article
-> normalize
-> split into sections
-> split into paragraphs
-> split into sentences
-> remove reference sections
-> remove/truncate citation tails
-> detect context-dependent starts
-> merge with previous paragraph if needed
-> reject very short orphan fragments
-> assign good/bad/maybe labels
-> train Good/Bad n-gram models
For chunk construction:
clean paragraphs
-> merge small adjacent paragraphs from same article
-> add EOS at article boundary
-> pack to max_length
-> avoid cutting inside sentence when possible
16. Suggested thresholds
These are only starting points.
| Rule | Possible starting point |
|---|---|
| too short | fewer than 20-30 Persian tokens |
| too digit-heavy | digit ratio above 15-20% |
| too Latin-heavy | Latin ratio above 20-30%, unless technical text |
| orphan start | starts with pronoun/context marker and very short |
| low Persian ratio | reject unless intended mixed text |
| too repetitive | repeated line or repeated n-gram |
| suspicious tail | truncate after reference-like pattern |
Do not trust the thresholds blindly.
Sample accepted/rejected data and adjust.
17. How much orphaned data is harmful?
If rare, probably not a disaster.
Large corpora always contain some imperfect examples.
But if many chunks are orphaned fragments, the model may learn:
- incomplete starts
- weak local coherence
- continuation from missing context
- strange discourse transitions
- bad paragraph openings
For a small 0.8B model, I would be stricter than for a huge model.
Small models have less capacity to average away noise.
So:
For Good CPT corpus:
be conservative.
For broad noisy corpus:
maybe keep some.
For final high-quality Persian corpus:
reject or merge orphan fragments.
18. Bottom line
For /10:
Orphaned pronouns and context-dependent chunks are not catastrophic if rare, but they are not high-quality CPT examples. Prefer article/section/paragraph-aware chunking, merge short context-dependent chunks with previous context, use EOS between documents, and reject very short orphan fragments.
For /11:
Good/Bad data should be defined by imitation target, not just surface cleanliness. Good = text you want the model to imitate. Bad = text you want your filter to detect and remove.
The simplest rule is:
Good data should be locally coherent Persian prose.
Bad data should represent the patterns you want to eliminate.
Discussion in the ATmosphere