{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreibo6mvew3v5ewwty27zirafpt76j3tna2hlldxm2bztmgp653tto4",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mnqro6kz74x2"
  },
  "path": "/t/custom-semantic-representation-bryla-beats-raw-text-in-24-27-configs-built-solo-on-an-rtx-2060-looking-for-feedback/176542#post_6",
  "publishedAt": "2026-06-08T02:16:14.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "FiLM",
    "Feature-wise transformations — Distill",
    "Prefix-Tuning",
    "Prompt Tuning",
    "MeCo"
  ],
  "textContent": "Hm. If we also merge in KnackAU’s approach, maybe the next step could be something like this:\n\n* * *\n\nThis looks like the sharpest version of the experiment so far.\n\nThe important part is not just that `BRYLA` won. The important part is the sequence:\n\n  1. the first control result was uncomfortable;\n  2. `BRYLA ≈ SHUFFLED` showed that the test was not really measuring structure;\n  3. the reason was diagnosed: values were self-identifying;\n  4. the task was redesigned with overlapping values;\n  5. after that, `BRYLA` separated clearly from `SHUFFLED`, while `RAW` and `RANDOM` stayed near chance.\n\n\n\nThat is a much stronger result than the earlier “Bryła beats RAW” story.\n\n## 1. Why the redesigned test is much better\n\nThe first version had a hidden flaw:\n\n\n    unique values per wall -> wall labels become redundant\n\n\nIf `formal` only ever belongs to `REG`, and `negative` only ever belongs to `POL`, then the model does not need the structure. It only needs the set of values.\n\nSo this result:\n\n\n    BRYLA    100.0%\n    SHUFFLED  99.9%\n\n\nwas actually the key diagnostic signal. It showed that the model could solve the task from value identity alone.\n\nThe redesigned version is much better:\n\n\n    all walls share A/B/C\n    target = hidden wall\n    cel = (S1 == S2)\n\n\nNow the model must know which value belongs to which wall. A bag of values is no longer enough.\n\nThat makes the new result meaningful:\n\n\n    RAW/RANDOM ≈ chance\n    SHUFFLED   ≈ 79–81%\n    BRYLA      = 100%\n\n\nMy interpretation:\n\n> The new test finally creates a condition where wall assignment matters. That is the right kind of synthetic test for Bryła.\n\n## 2. How I would state the current conclusion\n\nI would still keep the claim conditional, but it is now much stronger:\n\n> In the controlled overlapping-values setup, Bryła is not merely adding prefix format, domain metadata, or copyable values. It carries useful information through correct field/value assignment.\n\nBut I would not yet claim:\n\n> Bryła generally beats raw text on natural Polish data.\n\nThe current result shows:\n\n\n    structure can matter\n    and\n    when structure matters\n\n\nThe next step is to test whether natural Polish data has enough of this kind of ambiguity.\n\nNatural language probably does contain many overlapping-value cases:\n\nValue | Possible meanings depending on field\n---|---\n`high` | urgency, certainty, intensity, importance\n`negative` | sentiment, polarity, negation, evaluation\n`reported` | source type, evidentiality, discourse status\n`formal` | register, document type, communication style\n`core` | salience, topic focus, answer relevance\n\nSo the redesigned synthetic test is not arbitrary. It is a simplified version of a real problem.\n\n## 3. Where KnackAU’s suggestion fits\n\nI think KnackAU’s “fewer floats, no token tax” idea should not be treated as a competing direction.\n\nIt is better seen as another carrier for the same structure:\n\n\n    Bryła structure -> token serialization -> model\n    Bryła structure -> projected vector / conditioning signal -> model\n\n\nSo the core question becomes:\n\n> Is Bryła useful because it is a token sequence, or because it is structured side information?\n\nThose are different hypotheses.\n\nThe token form has advantages:\n\nToken Bryła | Pros | Cons\n---|---|---\n`[S1:A] [S2:B] ...` | readable, debuggable, easy to ablate | token tax, possible copying shortcuts, sequence-order sensitivity\n\nThe vector form has different advantages:\n\nVector Bryła | Pros | Cons\n---|---|---\nprojected signature vector | no token tax, compact, closer to latent conditioning | less interpretable, implementation details matter, may collapse assignments if designed poorly\n\nThis connects to several existing ideas:\n\n  * FiLM: feature-wise linear modulation; conditioning information changes intermediate features.\n  * Feature-wise transformations — Distill: clear explanation of how conditioning can be injected into neural networks.\n  * Prefix-Tuning: continuous task-specific vectors act like virtual prefix tokens.\n  * Prompt Tuning: soft prompts condition models without discrete text prompts.\n  * MeCo: metadata conditioning during training, followed by cooldown without metadata.\n\n\n\nThese are not identical to Bryła, but they support the general idea that side information can be passed either as tokens or as continuous conditioning.\n\n## 4. Suggested next experiment: token Bryła vs vector Bryła\n\nI would run the next comparison on the exact redesigned overlapping-values task, not on an easier clean set.\n\nSuggested conditions:\n\n\n    RAW\n    BRYLA-token\n    BRYLA-vector\n    BRYLA-token+vector\n    SHUFFLED-token\n    SHUFFLED-vector\n    RANDOM-vector\n\n\nThe reason to use the overlapping-values task is simple:\n\n\n    if the value itself identifies the wall, the test is too easy\n    if values overlap across walls, assignment becomes necessary\n\n\n## 5. How to interpret the outcomes\n\nResult | Interpretation\n---|---\n`BRYLA-vector ≈ BRYLA-token` | vector form may preserve the useful structure while avoiding token tax\n`BRYLA-vector > BRYLA-token` | structure may be better injected as latent conditioning than as text\n`BRYLA-token > BRYLA-vector` | small models may learn explicit symbolic tokens more easily\n`BRYLA-token+vector > both` | token and vector channels may be complementary\n`SHUFFLED-vector` high | vector representation may be preserving value set but losing assignment\n`RANDOM-vector` near chance | good; vector channel is not just a generic regularizer\n`RAW` near chance | good; target is not recoverable from text alone\n\nThe most important comparison is:\n\n\n    BRYLA-vector vs SHUFFLED-vector\n\n\nIf vector Bryła beats shuffled-vector on the overlapping-values task, then the vector really preserves assignment structure.\n\n## 6. Implementation variants worth separating\n\n“Vector Bryła” can mean several different things. I would separate them if possible:\n\nVariant | Description | Why test it\n---|---|---\nprojected global vector | encode all fields into one vector | closest to “fewer floats”\nBOS addition | add vector to first token embedding | simple implementation\nprefix virtual token | convert vector into one or more learned prefix embeddings | close to prefix-tuning\nFiLM-style modulation | use vector to scale/shift hidden features | stronger conditioning mechanism\ntoken + vector | keep readable tokens and add vector side channel | tests complementarity\n\nA minimal first version could be:\n\n\n    BRYLA-token\n    BRYLA-one-vector\n    BRYLA-token+one-vector\n    SHUFFLED-one-vector\n    RANDOM-one-vector\n\n\nThat is already enough to answer the main question.\n\n## 7. Metrics I would report\n\nFor the synthetic overlapping-values test:\n\nMetric | Why\n---|---\naccuracy | main task score\nchance baseline | sanity check\ntoken count | shows token-tax advantage\ntrain time | important on small hardware\ninference time | practical cost\nseeds | stability\nshuffled gap | measures structure sensitivity\n\nFor natural data later:\n\nMetric | Why\n---|---\nclean target-only PPL | avoids easy-tag artifact\ntask metric / F1 / chrF | checks actual output quality\nparser `OTHER%` | measures parser coverage\nfield entropy | detects dead/default fields\nvalue-overlap ratio | measures whether structure is actually needed\ntoken cost | practical tradeoff\n\n## 8. Bridge to natural data\n\nAfter token-vs-vector is tested on the overlapping-values setup, I would move to a small natural dataset.\n\nNot huge. Something like:\n\n\n    6 domains × 50 examples = 300 examples\n\n\nSuggested domains:\n\n\n    technical\n    geography\n    biography\n    science\n    daily life\n    sports/events\n\n\nRun:\n\n\n    RAW\n    DOMAIN\n    BRYLA-token\n    BRYLA-vector\n    SHUFFLED-BRYLA\n    RANDOM-BRYLA\n\n\nThe key question:\n\n\n    Does Bryła beat DOMAIN and SHUFFLED on natural data?\n\n\nIf yes, the result becomes much stronger.\n\nIf no, it is still informative:\n\nOutcome | Meaning\n---|---\n`DOMAIN ≈ BRYLA` | Bryła may mostly encode domain/topic\n`SHUFFLED ≈ BRYLA` | field assignment may not matter enough in that data\n`VECTOR ≈ TOKEN` | vector may remove token tax\n`TOKEN > VECTOR` | explicit symbolic representation may be easier for small models\n`VECTOR > TOKEN` | latent conditioning may be the better interface\n\n## 9. One extra diagnostic: value-overlap ratio\n\nThe most useful new diagnostic from this update is probably:\n\n\n    value-overlap ratio\n\n\nRough idea:\n\n\n    How often can the same value occur in multiple fields/walls?\n\n\nIf values are field-unique, structure is less necessary.\n\nIf values overlap across fields, assignment matters.\n\nSo I would report:\n\n\n    unique values per field\n    shared values across fields\n    field entropy\n    default ratio\n    assignment dependency\n\n\nThis directly connects the synthetic result to natural language.\n\n## 10. Suggested short conclusion\n\nMy current read:\n\n> The redesigned overlapping-values task is a real improvement because it removes the self-identifying-value shortcut. KnackAU’s vector idea is also worth integrating, because it tests whether Bryła’s useful part is the token serialization or the underlying structured signal. The next best experiment is probably token Bryła vs projected-vector Bryła on the same overlapping-values task, with shuffled-vector and random-vector controls.\n\nIf that works, then the natural next stage is:\n\n\n    small natural Polish dataset\n    + DOMAIN control\n    + SHUFFLED control\n    + RANDOM control\n    + token vs vector Bryła\n    + clean target-only metrics\n    + parser coverage\n\n\nThat would make the result much harder to dismiss.",
  "title": "Custom semantic representation (\"bryła\") beats raw text in 24/27 configs — built solo on an RTX 2060, looking for feedback"
}