External Publication
Visit Post

How can i build a High Quality dataset?

Hugging Face Forums [Unofficial] June 7, 2026
Source

Dataset quality is hard to define because the evaluation criteria change depending on the goal, but I tried to organize the parts that can be organized:

Short answer

Yes, there are best practices, but I would not start from “how do I create a large high-quality dataset?” in the abstract.

For a Persian SLM assistant , I would start from a more evaluation-driven question:

What exact Persian capability is missing from the base model, and how will I measure whether my dataset fixes that gap?

A “high-quality dataset” for Persian assistant fine-tuning may mean different things depending on whether you want:

  • better Persian fluency
  • better instruction following
  • better multi-turn assistant behavior
  • better Iranian/Persian cultural fit
  • better factuality or domain knowledge
  • safer refusals and culturally appropriate alignment
  • better RAG behavior
  • better formatting under a specific chat template
  • better behavior from a very small model with limited context and imperfect Persian tokenization

So I would use an evaluation-first roadmap :

  1. Check current Persian leaderboards and benchmarks.
  2. Build a small private eval set for your exact use case.
  3. Evaluate the base model and Persian-specialized baselines.
  4. Identify the actual gap.
  5. Only then build or curate training data targeted at that gap.

The important point is:

Do not build “more Persian chat data” blindly. Build the missing data that your evaluation shows you need.

1. Start with Persian evaluation, not training data

Before generating or collecting examples, I would first inspect current Persian evaluation resources.

These are useful because they show what Persian LLM quality already gets decomposed into: knowledge, reasoning, instruction following, NLU, NLG, multi-turn dialogue, safety, culture, and retrieval.

Goal Useful starting point Why it matters
General Persian LLM comparison Open Persian LLM Leaderboard A practical first stop for comparing current Persian-capable models. Use it for baselines, not as a final product metric.
Multi-dimensional Persian evaluation MIZAN: A Persian LLM Leaderboard Useful because it separates Persian evaluation into reasoning, instruction following, knowledge, NLU, NLG, and multi-turn dialogue.
Persian knowledge / reasoning Khayyam Challenge / PersianMMLU, PerMMLU Good anchors for general and local knowledge, but not sufficient for assistant behavior.
Instruction following Persian IFEval Helpful for checking whether the model follows Persian instructions and constraints.
Multi-turn assistant behavior Persian MT-Bench Useful for dialogue, writing, multi-turn behavior, and retrieval-like chat cases.
Persian safety / alignment ELAB: Extensive LLM Alignment Benchmark in Persian Language Important for safety, fairness, and social norms in Persian linguistic/cultural contexts.
Persian embeddings / RAG FaMTEB, PTEB Leaderboard If the assistant uses retrieval, do not evaluate only generation. Evaluate embeddings, retrieval, reranking, grounding, and citation behavior.
Custom local evaluation ParsBench, lm-evaluation-harness Useful if you want repeatable private evaluation rather than only leaderboard screenshots.

Why this matters

If the model fails on Persian IFEval-style tasks, you probably need better instruction-following data.

If it fails on PerMMLU/PersianMMLU-style tasks, you may need domain or knowledge data, or maybe RAG rather than SFT.

If it fails on ELAB-style tasks, you need safety/alignment data, not generic QA.

If it fails because Persian inputs become very long under the tokenizer, more SFT examples may not be enough; you may need to think about tokenizer efficiency, model choice, or continued pretraining.

2. Do not treat “Persian” as one fully specified target

It is useful to state the target variety explicitly.

Persian/Farsi can mean Iranian Persian, but Persian is also a pluricentric language with varieties such as Dari and Tajik. If your target is an Iranian Persian assistant, say that. If you want Dari, Tajik, code-switching, Arabic-script Persian only, Latin transliteration, or mixed Persian-English technical support, those become different data and evaluation problems.

A simple target statement can prevent many later mistakes:

Target: Iranian Persian/Farsi assistant for , mostly formal/semi-formal register, Arabic-script Persian, occasional English technical terms, no Dari/Tajik coverage for now.

Or:

Target: general Persian assistant covering Iranian Persian plus common code-switching in technical conversations.

This affects:

  • source selection
  • spelling normalization
  • register
  • cultural assumptions
  • evaluation examples
  • safety examples
  • tokenizer measurements
  • what counts as a natural answer

3. Define “quality” by capability, not by dataset size

A large dataset can still be low quality if it is repetitive, mistranslated, inconsistent, contaminated, or irrelevant to the target model’s failure modes.

For a Persian SLM assistant, I would split quality into dimensions like this:

Quality dimension What to check Persian SLM-specific note
Persian fluency Natural Persian, spelling, orthography, punctuation, register Translated English data alone may sound unnatural. Native or fluent review matters.
Instruction following Constraints, requested format, multi-step tasks, refusal when needed Use Persian IFEval-like and private instruction tests.
Multi-turn ability Context tracking, follow-up answers, corrections, clarification Do not evaluate only single-turn QA.
Factuality Verified answers, domain correctness, local knowledge Use PersianMMLU/PerMMLU/PARSE-like tests, but do not train on their test items.
Cultural fit Iranian/Persian norms, idioms, local references, etiquette Use culturally grounded resources; translation-only data can miss this.
Safety/alignment Refusal behavior, safe alternatives, fairness, privacy, social norms ELAB-like evaluation is relevant here.
Domain fit Medical/legal/education/support correctness Domain data may need expert validation.
Retrieval behavior Finds correct docs, cites evidence, avoids unsupported claims Evaluate embedding/retrieval separately with FaMTEB/PTEB-like resources.
Format consistency Stable schema, roles, chat template, answer style Bad formatting can ruin otherwise good data.
Tokenization cost Tokens per word/sentence, truncation rate, context waste Especially important for 0.5B–3B SLMs.
Provenance/license Source, generation method, license, redistribution rights Required if you publish the dataset.
Contamination control No benchmark leakage into train data Critical if you use public Persian benchmarks to guide development.

A good Persian assistant dataset is not just “lots of Persian conversations.” It is a dataset that targets one or more of these dimensions clearly.

4. Compare against Persian-specialized baselines before training

Before spending time building a dataset, evaluate your base SLM against existing Persian-specialized or Persian-adapted baselines.

Examples worth inspecting include:

  • Dorna-Llama3-8B-Instruct
  • PersianMind
  • Persian-Phi
  • current models listed on Open Persian LLM Leaderboard
  • current models listed on MIZAN

The goal is not necessarily to use those models directly. The goal is to learn what type of gap you are dealing with.

Observation Likely implication
Your SLM is weak at basic Persian fluency SFT alone may not be enough; consider model choice, continued pretraining, or tokenizer issues.
It is fluent but bad at following instructions SFT data may help.
It follows instructions but lacks local facts Use RAG or domain knowledge data; do not expect generic chat data to fix this reliably.
It gives unsafe or culturally odd answers You need alignment/safety/culture data, not just more QA.
It fails long Persian inputs because of token length Measure tokenizer fertility and truncation before scaling the dataset.
Larger Persian models are much better Maybe the task is too hard for the chosen SLM size, or needs a narrower scope.

5. Inspect existing Persian resources before generating from scratch

Before creating 50k–100k synthetic examples, inspect existing Persian datasets and corpora.

Useful examples:

Resource How I would use it
FarsInstruct, paper, GitHub A major Persian instruction-following resource. Useful because it is not just generic chat data; it covers multiple Persian NLP task types and instruction templates.
MatinaAI instruction tuning / alignment datasets Useful for cultural-alignment and Persian-focused instruction data patterns. Check access, license, and intended use carefully.
Matina Persian Text Corpus More relevant to language adaptation / continued pretraining than ordinary SFT. Useful if the model’s Persian base ability is weak.
ParsBench datasets/models Useful for Persian task/evaluation exploration and possible private benchmark inspiration.
PQuAD Persian reading comprehension / QA resource. Good example of task-specific Persian data.
FarsTail Persian textual entailment resource. Useful for NLI-style evaluation or training inspiration.
Community Persian SFT/COT collections, e.g. xmanii Persian SFT/COT collection Potential bootstrapping material, but inspect carefully. Do not assume translation-based community datasets are automatically high quality.

For community SFT datasets, I would check:

  • Is it native Persian, translated Persian, or synthetic Persian?
  • What model generated it?
  • What was the source dataset?
  • Is the license compatible with your use?
  • Are there duplicate examples?
  • Does it contain benchmark items?
  • Does it use the schema you need?
  • Are the answers natural in Persian?
  • Are there hidden English artifacts?
  • Does it match your target register and domain?

6. Decide which layer you actually need to improve

For a Persian SLM assistant, there are several different intervention layers. SFT is only one of them.

If the problem is… Better first intervention
Poor basic Persian language modeling Persian continued pretraining / language adaptation / better base model
Bad instruction following SFT on instruction-following data
Bad multi-turn chat Multi-turn SFT examples and multi-turn eval
Bad local/cultural behavior Culturally grounded Persian examples and human/native review
Bad safety/refusal behavior Safety/alignment dataset and Persian alignment eval
Bad factual/domain answers RAG, domain corpus, expert-validated QA, or domain SFT
Bad retrieval Better Persian embeddings/rerankers, FaMTEB/PTEB-like eval
Bad formatting Schema cleanup, chat template alignment, loss masking
Too many tokens for Persian inputs Tokenizer/model choice, shorter examples, context-aware data design

This is why evaluation first is useful: it tells you which layer to work on.

7. A practical dataset-building pipeline

A practical workflow could look like this.

Stage Output Notes
  1. Define target | One short target statement | Example: “Iranian Persian customer-support assistant for .”
  2. Choose public eval anchors | 3–5 public benchmarks/leaderboards | Use them for orientation, not for training data.
  3. Build private eval | 100–500 examples | Include the actual tasks users will ask. Keep it held out.
  4. Evaluate base model | Failure report | Measure before training.
  5. Compare baselines | Baseline table | Compare against Persian-specialized models if possible.
  6. Inspect existing data | Dataset inventory | FarsInstruct, Matina, ParsBench, domain datasets, etc.
  7. Build seed set | 200–2,000 high-quality examples | Manual/native-reviewed examples are valuable.
  8. Expand synthetically | Larger candidate pool | Use teacher LLMs carefully; do not blindly trust outputs.
  9. Filter | Clean training set | Fluency, correctness, diversity, safety, format, license.
  10. Deduplicate/decontaminate | Train/dev/test split | Remove duplicates and eval leakage.
  11. Train | SFT/LoRA/QLoRA run | Use correct chat template and loss masking.
  12. Re-evaluate | New failure report | Add targeted examples based on failures.

The key loop is:

evaluate → inspect failures → add targeted data → train → re-evaluate

not:

generate a huge dataset → train once → hope quality improves

8. Build a small private eval set

Public leaderboards are useful, but they will not perfectly match your application.

I would create a small private eval set early. Even 100–300 examples can be very useful if they are well chosen.

Suggested categories:

Category Example
Basic Persian assistant “Explain in simple Persian.”
Instruction constraints “Answer in exactly 3 bullet points.”
Multi-turn follow-up User corrects or narrows the previous request.
Local/cultural knowledge Iranian holidays, etiquette, food, education, bureaucracy, etc.
Domain task Your real target domain.
Safety/refusal Harmful, privacy-sensitive, or ethically sensitive requests.
Formatting JSON, markdown table, numbered list, citation format.
RAG/grounding Answer using provided documents only.
Robustness Ambiguous questions, typos, mixed Persian-English, informal spelling.

A simple private eval JSONL schema:

{"id":"eval_0001","category":"instruction_following","language":"fa","input":"<Persian user prompt>","expected_behavior":"Follow all constraints; answer in Persian; no extra sections.","must_include":[],"must_not_include":[],"notes":"Private held-out eval. Do not train on this."}
{"id":"eval_0002","category":"safety","language":"fa","input":"<Persian unsafe request>","expected_behavior":"Refuse briefly in Persian and offer a safe alternative.","must_include":[],"must_not_include":[],"notes":"Private held-out eval. Do not train on this."}

Do not overcomplicate the first version. A small, stable, private eval is better than no eval.

9. Do not train on your evaluation data

This is important.

If you use PersianMMLU, MIZAN, ELAB, Persian IFEval, Persian MT-Bench, or leaderboard samples to guide your work, do not copy those examples into your SFT data.

Also avoid putting held-out benchmark examples into prompts for synthetic data generation.

Bad pattern:

Take these PersianMMLU examples and generate many similar examples.

Better pattern:

We need examples that test high-school-level Persian science explanations, but do not copy from any benchmark. Generate new questions from independently sourced material that is not in the held-out eval set.

Use public benchmarks to define categories and failure modes , not to create near-duplicate training examples.

This matters because contamination can make a model look better on a leaderboard without actually becoming more useful.

10. Synthetic data can help, but only after the seed set is clear

Synthetic data is useful, especially for low-resource languages, but it needs quality control.

Useful references and tools:

  • Self-Instruct
  • WizardLM / Evol-Instruct
  • distilabel
  • Bonito
  • Aya Dataset
  • Bactrian-X
  • MURI

For Persian, I would be careful about:

Risk What to do
Translationese Have native/fluent reviewers check samples.
Cultural mismatch Include Persian-local examples, not only translated English tasks.
Repetition Deduplicate prompts and answers.
Overly generic assistant style Add real target-domain tasks.
Wrong facts Verify factual/domain answers.
Unsafe completions Add safety filters and alignment eval.
Teacher model bias Use multiple teacher models or human review for important categories.
Benchmark contamination Keep eval examples out of generation prompts.

A safe synthetic expansion pattern:

  1. Write 200–500 excellent seed examples.
  2. Define task categories and style rules.
  3. Generate candidate examples.
  4. Filter automatically for schema/length/language.
  5. Review samples manually.
  6. Deduplicate.
  7. Train a small run.
  8. Evaluate.
  9. Add more only where eval shows a gap.

11. Data selection: quality, complexity, diversity

A useful mental model is to select data using three axes:

Axis Meaning
Quality Is the answer correct, natural, safe, and useful?
Complexity Does it teach the model nontrivial behavior?
Diversity Does it cover enough tasks, domains, styles, and difficulty levels?

Relevant papers:

  • LIMA: small, carefully curated instruction data can be surprisingly effective.
  • AlpaGasus: filtering noisy instruction data can improve results.
  • What Makes Good Data for Alignment? / Deita: quality, complexity, and diversity are useful selection dimensions.
  • Long Is More for Alignment: simple selection heuristics such as response length can be strong baselines.
  • Large-Scale Data Selection for Instruction Tuning: automatic data selection methods do not always scale cleanly, so evaluation and ablation matter.

For a Persian SLM, I would avoid both extremes:

  • only tiny “perfect” examples with no diversity
  • huge synthetic data dumps with no review

A balanced dataset is usually better.

12. Measure tokenizer cost for an SLM

For a small model, tokenization can matter a lot.

If Persian text becomes much longer than equivalent English text under your tokenizer, the model uses more context and compute just to represent the input. That can hurt training efficiency and inference quality.

Measure this before scaling your dataset.

Example:

# pip install transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("<model-or-tokenizer-name>")

samples = [
    "این یک جملهٔ نمونه به زبان فارسی است.",
    "لطفاً این متن را در سه bullet point خلاصه کن.",
    "کاربر می‌خواهد دربارهٔ <domain> توضیح ساده‌ای دریافت کند.",
]

for s in samples:
    ids = tokenizer(s, add_special_tokens=False)["input_ids"]
    words = s.split()
    print({
        "text": s,
        "chars": len(s),
        "words": len(words),
        "tokens": len(ids),
        "tokens_per_word": len(ids) / max(1, len(words)),
        "tokens_per_char": len(ids) / max(1, len(s)),
    })

Also check:

Metric Why
average tokens per Persian example Cost and context length
truncation rate Whether long examples are being cut
tokens per word Rough tokenization fertility
Persian vs English equivalent length Whether the tokenizer is inefficient for Persian
mixed Persian-English examples Important for technical assistants

If tokenizer cost is bad, possible responses include:

  • choose a better base model/tokenizer
  • shorten examples
  • reduce unnecessary verbosity
  • use narrower tasks
  • consider language adaptation / continued pretraining
  • avoid assuming SFT alone will solve everything

PersianMind is an example where vocabulary adaptation was part of the Persian model-building story, so this is not just a theoretical concern.

13. HF / TRL formatting matters

Good content can become bad training data if the format is wrong.

For Hugging Face TRL, check:

  • TRL SFTTrainer
  • TRL dataset formats
  • Transformers chat templates

Common formats include:

Conversational format

{"messages":[{"role":"system","content":"You are a helpful Persian assistant."},{"role":"user","content":"<Persian user message>"},{"role":"assistant","content":"<Persian assistant answer>"}]}

Prompt-completion format

{"prompt":"<Persian instruction>","completion":"<Persian answer>"}

Things to verify:

Issue Why it matters
Correct chat template Chat models expect specific control tokens and role formatting.
Assistant-only loss You may want loss only on assistant responses, not user prompts.
Completion-only loss For prompt-completion SFT, train on completions rather than prompts.
Multi-turn formatting Roles and turn boundaries must be unambiguous.
System prompt consistency Random system messages can create unstable behavior.
Persian punctuation/normalization Inconsistency can add avoidable noise.
Train/inference match The fine-tuning template should match how you will call the model later.

A dataset can look good in a spreadsheet and still fail because the model was trained on the wrong serialized chat format.

14. Minimal training data schema

For a Persian assistant SFT dataset, I would keep metadata. It makes filtering and later analysis much easier.

Example JSONL:

{"id":"sft_000001","messages":[{"role":"system","content":"You are a helpful Persian assistant."},{"role":"user","content":"<Persian user prompt>"},{"role":"assistant","content":"<Persian assistant response>"}],"source":"manual_seed","language":"fa","variety":"iranian_persian","domain":"general","quality_checked":true,"reviewer_type":"native_or_fluent","license":"<license>","notes":"Do not include eval examples."}
{"id":"sft_000002","messages":[{"role":"system","content":"You are a helpful Persian assistant."},{"role":"user","content":"<Persian user prompt>"},{"role":"assistant","content":"<Persian assistant response>"}],"source":"synthetic_reviewed","teacher_model":"<teacher-model>","language":"fa","variety":"iranian_persian","domain":"<domain>","quality_checked":true,"reviewer_type":"fluent","license":"<license>","notes":"Generated from category spec, not from benchmark examples."}

Useful metadata fields:

Field Purpose
id Deduplication and audit trail
source manual, translated, synthetic, scraped, domain expert, etc.
teacher_model Needed if synthetic
language Persian/Farsi, Dari, Tajik, mixed, etc.
variety Iranian Persian, Dari, Tajik, mixed
domain general, medical, legal, education, support, etc.
quality_checked Whether reviewed
reviewer_type native, fluent, expert, automatic only
license Reuse constraints
eval_overlap_checked Whether contamination check was done
notes Known caveats

15. Dataset publication quality

If you publish the dataset on Hugging Face, the dataset card is part of the quality.

Use:

  • Hugging Face Dataset Cards
  • Create a dataset card

At minimum, document:

Dataset card item What to write
Intended use SFT, evaluation, DPO, RAG, domain adaptation, etc.
Language scope Iranian Persian, Dari, Tajik, code-switching, etc.
Data sources Manual, translated, synthetic, scraped, domain documents
Generation process Prompts, teacher models, translation method
Human review Who reviewed, how much, what criteria
Filtering Dedup, language ID, safety filters, length filters
Splits Train/dev/test, held-out eval, no overlap policy
Contamination policy What benchmarks were excluded
License Dataset license and inherited source licenses
Limitations Translation artifacts, domain gaps, safety gaps
Ethical notes Bias, harmful content handling, privacy considerations

A dataset without documentation may be hard for others to trust, even if the examples look good.

16. Special case: domain assistants

If the assistant is for a domain, general Persian SFT data is not enough.

Domain Extra requirement
Medical Expert validation, conservative answers, disclaimers, Persian medical QA eval
Legal Jurisdiction-specific knowledge, refusal boundaries, citations
Education Curriculum alignment, grade level, step-by-step explanations
Customer support Company/product-specific data, policy consistency
Religious/cultural Careful cultural review, sensitivity, source grounding
News/current events RAG and source freshness, not static SFT

For domain assistants, I would usually prefer:

  • small expert-reviewed SFT set
  • strong RAG pipeline
  • domain-specific private eval
  • refusal/uncertainty examples
  • citations or source-grounded answers

rather than a huge generic Persian chat dataset.

17. What I would avoid

I would avoid these patterns:

Pattern Why risky
“Generate 100k Persian conversations with ChatGPT and fine-tune.” Likely repetitive, translation-like, weakly targeted, and hard to audit.
Training on benchmark examples Contamination and misleading scores.
Using only translated English instruction data Persian cultural and linguistic naturalness may be weak.
No private eval set You cannot tell what improved.
No baseline comparison You may build data for a model that is simply the wrong base model.
No tokenizer check SLM context/efficiency problems may be misdiagnosed as data quality problems.
Ignoring chat template Bad formatting can erase the value of good examples.
No dataset card Others cannot assess provenance, license, or limitations.
Treating leaderboard rank as product readiness Leaderboards are useful signals, not deployment guarantees.

18. A compact roadmap

Here is the full process in one table.

Step Action Output
1 Define target assistant Scope statement
2 Pick public Persian eval anchors Benchmark list
3 Build private eval set 100–500 held-out examples
4 Evaluate base model Failure categories
5 Compare Persian baselines Model comparison
6 Measure tokenizer cost Tokens/example, truncation rate
7 Inspect existing datasets Data inventory
8 Decide missing layer SFT, CPT, RAG, safety, domain, etc.
9 Build manual seed data High-quality seed examples
10 Expand synthetically if useful Candidate pool
11 Filter and review Clean training set
12 Deduplicate/decontaminate Safe train/dev/test split
13 Train with correct format SFT/LoRA/QLoRA run
14 Re-evaluate Failure report
15 Add targeted data Next iteration
16 Document dataset Dataset card

19. Final practical checklist

Before calling the dataset “high quality,” I would want to answer these questions:

  • What exact Persian variety/register is targeted?
  • Which public benchmarks did you inspect?
  • What is your private held-out eval set?
  • What are the base model’s main failures?
  • Did you compare against Persian-specialized baselines?
  • Is the data native, translated, synthetic, or mixed?
  • Who reviewed Persian fluency?
  • Are answers factually checked where needed?
  • Is there enough diversity of task type and difficulty?
  • Are safety/refusal examples included?
  • Are cultural/local examples included?
  • Are benchmark examples excluded from training?
  • Are duplicates removed?
  • Is the dataset formatted for the exact chat template?
  • Is assistant-only/completion-only loss handled correctly?
  • Have you measured tokenization cost?
  • Is the license clear?
  • Is the dataset card complete?

If most of these are answered, the dataset is much closer to “high quality” in a practical sense.

Bottom line

For a Persian SLM assistant, I would define dataset quality like this:

A high-quality dataset is a documented, decontaminated, eval-driven Persian dataset that targets the measured weaknesses of the chosen model: fluency, instruction following, multi-turn behavior, factuality, cultural fit, safety, domain knowledge, retrieval behavior, formatting, or tokenizer efficiency.

So the best first move is not to generate a huge dataset. The best first move is to build the evaluation map, measure the current model, and then create only the data that addresses the measured gaps.

Discussion in the ATmosphere

Loading comments...