Weekend Research Update — SFT Experiments, BPB Metric & New Pretraining BEST

Project Inkblot | Mateusz Piesiak | 26–28 June 2026

A quick personal note before diving in — I genuinely wish I had more time to participate actively in the discussion here. Life has been quite full lately: day job as a Manufacturing Controller, family, kids, the household, and everything else that comes with being a solo researcher doing this entirely on the side. I read every comment carefully and they matter a lot to me — I just do not always get to respond as quickly as I would like. This weekend report is my way of making up for the silence. Thank you for your patience.

Pretraining Progress — New BEST Checkpoint

Since my last post the pretraining has continued on AWS g6.2xlarge (L4 24GB). Here is the full checkpoint history:

Step	Val Loss	PPL	Notes
30,000	4.0508	57.5	Cycle 2 start (expanded dataset)
45,000	3.8319	46.1	Previous BEST
50,000	3.7447	42.3
53,950	3.7447	42.3	Autoresume STOP
55,000	3.6835	39.8
60,000	3.6361	37.9
65,000	3.5968	~36.5	New BEST

The model continues to converge steadily. At step 65,000 we have processed approximately 4.26B tokens (65,000 steps × batch 4 × grad_accum 16 × seq_len 1024). Chinchilla optimum for 464M params is ~9.3B tokens so we are at 46% of the way.

Weekend SFT Experiments

This weekend I ran three SFT experiments. Here is exactly what happened, including the raw outputs.

Experiment 1 — Colab T4, 120 examples, freeze 10/16 layers

I started with the step 50,000 checkpoint (PPL 42.3) and a small dataset of 120 ChatML examples across 5 domains. Froze the embedding and layers 0–9, training only the top 6 layers. Used fp16 with gradient checkpointing on a T4 (15GB VRAM), 7 epochs at LR 3e-6.

The training curve looked promising on paper:

Ep 1/7  val=7.49  PPL=1796
Ep 2/7  val=5.79  PPL=326
Ep 3/7  val=4.57  PPL=96
Ep 4/7  val=3.77  PPL=43
Ep 5/7  val=3.35  PPL=28
Ep 6/7  val=3.15  PPL=23
Ep 7/7  val=3.07  PPL=21.6  ← saved

But the actual generation told a very different story:

Prompt: "Co to jest szlak mTOR i jaką rolę odgrywa w nowotworach?"

Output: 123466678910101011121314151718181819206070909090899090909190909094909090
        CCC C C C G G G H G G G M G G G A G G G I G G G I G G G I G G G I...

PPL 21.6 after 7 epochs was pure memorisation, not learning. The model reproduced numerical patterns (TPM values 1, 2, 3… 100, 200, 500, 1000) from the RNA-seq training examples. 120 examples × 7 epochs = 742 forward passes on the same data. Classic overfitting on a tiny dataset.

Experiment 2 — Colab T4, 120 examples, full unfreeze

Same 120 examples but this time all 16 layers unfrozen, 2 epochs.

Ep 1/2  val=7.49  PPL=1796
Ep 2/2  val=2.68  PPL=14.5  ← saved (interrupted)

Output:

Prompt: "Explain the mechanism of action of tyrosine kinase inhibitors in CML"

Output: /. / W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W
        W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W...

Catastrophic forgetting in 88 seconds. Training all 464M parameters on 120 examples erased 5.2B tokens of pretraining knowledge. The model replaced everything with a loop on a single token.

Experiment 3 — AWS L4, 4300 examples, structural MoE-aware freeze

Based on KnackAU’s earlier comment and my own architecture inspection, I moved from sequential layer freezing to structural freezing — freezing by component type rather than by layer index:

for name, param in model.named_parameters():
    freeze = (
        'token_emb' in name or
        any(f'layers.{i}.' in name for i in range(4)) or
        'router' in name or
        'routed_experts' in name or
        'shared_experts' in name or
        'shared_gate' in name
    )
    param.requires_grad = not freeze

The key decision here was freezing the MoE routers and experts. Why? Because the Expert Sparsity Analysis (done earlier this weekend) showed that the router learned domain-specific expert allocation with zero supervision:

Medical vs Python   overlap = 0.00  ← perfect domain specialisation
Medical vs Polish   overlap = 0.15  ← clear specialisation
Python  vs English  overlap = 0.11  ← clear specialisation

This emerged purely from pretraining on the mixed 4-domain corpus. SFT should not destroy this.

The split ended up at 278.6M frozen (embedding + first 4 layers + all MoE routing and experts) and 185.9M trainable (Attention QKV, Mamba SSM, Dense/SwiGLU, lm_head) — about 40% of the model.

The dataset was much larger this time: 4,300 ChatML examples total. 3,500 clinical reasoning examples with step-by-step analysis, 800 examples from MedDialog/MedQA/PubMedQA/ChatDoctor/iCliniq, and 50 of my own Polish medical examples covering RNA-seq omics and drug mechanisms.

Training curve:

Val BEFORE  9.92   PPL=20,257
Ep 1/3      train=0.46  val=0.060  PPL=1.1
Ep 2/3      train=0.04  val=0.042  PPL=1.0  ← BEST saved

Val loss 0.04 looked incredible but it was an artefact of data leakage. When building the dataset I mixed train_final.csv and val.csv into one pool and then split 95/5 randomly. The model was being evaluated on examples it had already seen during training. The actual generalisation is unknown.

The output confirmed the problem:

Prompt: "What is the mTOR signaling pathway and its role in cancer?"

Output: Guthrie struct char ******* Tibpectral vagus neutrophonsai Monaster
        sappveston CoQ provinandelion oracle alp powiatowszeostridochastic...

Root cause is the same as Experiments 1 and 2 — base model PPL 39.8 is simply too weak for SFT to produce anything useful. This confirms KnackAU’s point that Chinchilla is a checkpoint, not a ceiling. The SFT pipeline, dataset, and freeze strategy are all correct. The base model just needs more pretraining first.

Evaluation Results (step 50,000, before SFT)

MCQ Log-Likelihood

I ran a proper MCQ eval with full-text scoring and a BPE fix (the previous version had a merge bug where single-word continuations produced cont=0 tokens — fixed by stripping trailing space from prompt and including it with the continuation, which is the standard lm-evaluation-harness approach).

🟢 Easy    (5 questions)  3/5 (60%)  Paris ✅  print() ✅  Warszawa ✅
🟡 Medium  (5 questions)  2/5 (40%)  mTOR ✅  RNA-seq ✅
🔴 Hard    (5 questions)  0/5 (0%)   expected for base model pre-SFT

Tokenizer Compression — Titan 64K vs Mistral-7B vs GPT-2

Polish General    Titan 4.04 c/t  |  Mistral 2.47 (+67%)  |  GPT-2 1.93 (+110%)
Polish Medical    Titan 3.25 c/t  |  Mistral 2.30 (+41%)  |  GPT-2 2.07 (+57%)
English Tech      Titan 5.19 c/t  |  Mistral 4.84 (+7%)   |  GPT-2 5.05 (+3%)
Python Code       Titan 2.52 c/t  |  Mistral 2.77 (-9%)   |  GPT-2 2.22 (+14%)

The 64K Polish-optimised tokenizer gives +67% better compression for Polish text vs Mistral. This directly reduces compute cost for Polish-language inference.

Expert Sparsity — Router Specialisation across 4 MoE layers (L0/L4/L8/L12)

Medical (top-5)  E24:24  E29:23  E16:21  E23:21  E3:21
Python  (top-5)  E9:52   E28:50  E30:48  E27:47  E10:47
Polish  (top-5)  E24:26  E4:25   E30:23  E29:21  E22:21
English (top-5)  E23:18  E3:18   E30:15  E13:14  E29:14

Domain specialisation (IoU of top-5 expert sets)
  Medical  vs Python   overlap = 0.00  ✅ Perfect specialisation
  Medical  vs Polish   overlap = 0.15  ✅ Clear specialisation
  Python   vs English  overlap = 0.11  ✅ Clear specialisation
  Polish   vs English  overlap = 0.25  ✅ Clear specialisation

The router learned domain-specific expert allocation with zero supervision — this emerged purely from pretraining on the mixed 4-domain corpus.

BPB Evaluator — responding to KnackAU’s feedback

As KnackAU correctly pointed out, PPL is tokenizer-dependent and cannot fairly compare Titan v1 (50K vocab, PPL ~27.5) with Titan v2 (64K vocab, PPL ~36.5).

Bits Per Byte (BPB) is tokenizer-invariant:

BPB = total_NLL_nats / (ln(2) × n_bytes_UTF8)
    = log₂(PPL) / avg_bytes_per_token

Implementation:

@torch.no_grad()
def compute_bpb(model, tok, text: str) -> dict:
    n_bytes  = len(text.encode('utf-8'))
    ids      = tok.encode(text, add_special_tokens=False)

    input_ids = torch.tensor([ids]).to(DEVICE)
    out = model(input_ids)
    logits = (out[0] if isinstance(out, tuple) else out)[0]

    log_probs = F.log_softmax(logits[:-1].float(), dim=-1)
    targets   = input_ids[0, 1:]
    nll_per_token = -log_probs[range(len(targets)), targets]

    total_nll = nll_per_token.sum().item()
    bpb = total_nll / (math.log(2) * n_bytes)
    return {"bpb": bpb, "ppl": math.exp(nll_per_token.mean().item())}

The evaluator tests across 5 domains (English General, Polish General, Medical English, Medical Polish, Python Code). BPB results for v1 vs v2 comparison will be published once v2 reaches the target PPL.

Interpretation scale:

< 1.0    excellent (better than human for English)
1.0–1.5  good
1.5–2.0  average
> 2.0    undertrained

SFT Dataset — Ready for Next Run

The dataset for the next SFT attempt is built and waiting:

sft_master_v2.jsonl — 4,300 high-quality ChatML examples (7.6MB)

• 3,500 medical examples from train_final.csv — clinical reasoning with step-by-step analysis, system prompt “You are an expert medical AI assistant”
• 800 examples from val.csv — MedDialog, MedQA, PubMedQA, ChatDoctor, iCliniq, system prompt “You are an empathetic AI health companion”
• 50 of my own Polish medical examples — RNA-seq omics analysis, drug mechanism explanations (PL/EN), patient education in Polish

Validation split is 197 examples in a separate file (sft_val_v2.jsonl) with no mixing with train. Lesson learned.

SFT Architecture for Next Run

For the next SFT run (once pretraining reaches PPL ~30-32) the plan is the same structural freeze that was validated in Experiment 3:

# Structural freeze — preserves MoE domain specialisation
freeze = (
    'token_emb' in name or
    any(f'layers.{i}.' in name for i in range(4)) or
    'router' in name or
    'routed_experts' in name or
    'shared_experts' in name or
    'shared_gate' in name
)
# Frozen    278.6M (embedding + 4 layers + all MoE routing)
# Trainable 185.9M (Attention, Mamba, Dense, lm_head)

Target config is LR 3e-6, 3 epochs, GRAD_ACCUM 4, fp32 on L4.

What’s Next

Current   step 65,000  |  PPL 36.5  |  4.26B tokens
Target    step ~92,000  |  PPL ~33-34  |  ~6B tokens (early stop considered)
          step ~164,490  |  PPL ~30-32  |  9.3B tokens (Chinchilla optimum)

After pretraining reaches the target:

1. BPB eval — v1 vs v2 cross-tokenizer comparison
2. SFT with sft_master_v2.jsonl (4,300 examples, structural freeze)
3. Per-domain eval post-SFT
4. Architecture ablation at 57M scale (Mamba-only vs Attn+MoE vs triple hybrid)

Key Lessons This Weekend

1. SFT requires a competent base. PPL > 35 means garbage output regardless of dataset quality or freeze strategy. This is not a bug — it is physics.

2. MoE expert routing is precious. Freeze routers and experts during SFT. The domain specialisation (Medical vs Python overlap = 0.00) took billions of tokens to learn and can be destroyed in minutes.

3. Data leakage is subtle. Mixing sources before splitting produces optimistically low val loss that means nothing. Val set must be built separately before any train data is assembled.

4. BPB > PPL for cross-model comparison. Larger vocab mechanically inflates PPL. The models are more comparable than the raw PPL numbers suggest. *Update comming soon

5. 120 examples is not enough for 464M params. Minimum viable SFT for this model size is ~500-1000 high-quality instruction pairs. The 4,300-example dataset should work.

So yee that happen on weekend guys have nice reading hehe and yee Any Question Just Message me Happy to Answer For all Question @ProtopiaUk @KnackAU @vl-dev @John6666