{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreigymrhxasbfizatsmp3epcwxtjyl4ddzbadmo4xbsaas2movl4plq",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mnk2suwmuci2"
  },
  "path": "/t/custom-semantic-representation-bryla-beats-raw-text-in-24-27-configs-built-solo-on-an-rtx-2060-looking-for-feedback/176542#post_4",
  "publishedAt": "2026-06-05T10:58:27.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "@John6666"
  ],
  "textContent": "Thank you both — genuinely.\n\n@John6666 — your control ladder wasn’t just useful, it changed how I\ntest. This isn’t the first time your suggestions pushed me to a sharper\nexperiment, and each time the result got more trustworthy because of it.\nThe SHUFFLED / RANDOM / DOMAIN idea in particular saved me from publishing\na wrong conclusion. Thank you.\n\nAnd thank you to the second reply too — the “structure already lives in\nthe model’s latent geometry, the cure is fewer floats not more tokens”\nframing, plus the concrete signature-vector experiment, gave me a real\nvariant to measure. I added it (more below).\n\nHere’s what the controls actually showed — including where I was wrong.\n\n## I ran the controls. First result was uncomfortable.\n\nOn my balanced synthetic set, with the model predicting the SAME walls it\nreceived as input:\n\n\n    RAW       96.8%\n    DOMAIN    96.3%\n    BRYLA    100.0%\n    SHUFFLED  99.9%\n    RANDOM    96.2%\n\n\nBRYLA beats RAW, DOMAIN and RANDOM — good. But BRYLA ≈ SHUFFLED. That was\nthe important signal: shuffling which value belongs to which wall barely\nhurt. So the structure (the wall assignment) wasn’t being used — the model\nwas essentially copying the value set from input to output. The test was\nmeasuring reconstruction, not meaning. Exactly the “predicting easy tags”\ntrap John warned about.\n\n## Diagnosis: the values were self-identifying\n\nIn that setup each value was unique to one wall (“formal” only ever in\nREG, “negative” only in POL). So the model never needed the wall labels —\nthe value itself revealed its wall. A bag of values was enough. Shuffling\nlabels didn’t change the bag, so it didn’t hurt.\n\n## Redesigned test: hidden target + overlapping values\n\nTwo changes:\n\n  1. The model predicts a HIDDEN wall that is NOT in the input — so it must\ninfer, not copy.\n  2. All walls share the same value space (A/B/C), so a value no longer\nreveals its wall. Now the model MUST use the assignment.\n\n\n\nTarget rule: cel = (S1 == S2) — requires distinguishing S1 from S2.\nText carries only a neutral topic, never the target.\n\nResult (2700 pairs, balanced, with and without text — nearly identical):\n\n\n              chance   RAW    BRYLA   SHUFFLED   RANDOM\n    w/ text   66.7%   68.6%   100%    79.0%      66.4%\n    no text   66.7%   68.6%   100%    81.0%      68.6%\n\n\nNow SHUFFLED collapses ~20 points below BRYLA. RANDOM sits at chance.\nRAW sits at chance. So: the gain comes from real values in their correct\nwall positions — not from prefix format, not from text.\n\n## The actual conclusion (conditional, not triumphant)\n\nPutting both tests together:\n\n\n    unique values per wall   -> SHUFFLED = BRYLA  (structure redundant)\n    shared values per wall   -> SHUFFLED < BRYLA  (structure necessary)\n\n\nSo wall structure carries information exactly when values are NOT self-\nidentifying. And overlapping values are precisely the natural-language\ncase — “high” can be certainty, urgency, or intensity; “not” can be\nnegation or something else. My first synthetic set was simply too clean,\nwhich hid the value of structure. That’s the honest result.\n\n## On the signature-vector idea\n\nI also added a variant where the walls enter as ONE projected feature\nvector instead of ~28 tokens (the “fewer floats, no token tax” direction).\nOn the clean set it matched the token form. I want to re-run it on the\noverlapping-values task before claiming anything — that’s the harder case\nand the fair test. Will report back.\n\n## What I’m NOT claiming\n\nStill synthetic. This shows structure CAN matter and WHEN. It does not yet\nshow that my parser, on real Polish text, produces enough value-overlap\nfor structure to help in practice. That’s the next step you both pointed\nat: small natural dataset, clean target-only loss, parser coverage\nreported. Now I have a reason to invest in it — I know structure isn’t\nredundant a priori.\n\nThanks again. This thread moved the project further than the previous\nmonth did.",
  "title": "Custom semantic representation (\"bryła\") beats raw text in 24/27 configs — built solo on an RTX 2060, looking for feedback"
}