Custom semantic representation ("bryła") beats raw text in 24/27 configs — built solo on an RTX 2060, looking for feedback

Thank you both — genuinely.

@John6666 — your control ladder wasn’t just useful, it changed how I test. This isn’t the first time your suggestions pushed me to a sharper experiment, and each time the result got more trustworthy because of it. The SHUFFLED / RANDOM / DOMAIN idea in particular saved me from publishing a wrong conclusion. Thank you.

And thank you to the second reply too — the “structure already lives in the model’s latent geometry, the cure is fewer floats not more tokens” framing, plus the concrete signature-vector experiment, gave me a real variant to measure. I added it (more below).

Here’s what the controls actually showed — including where I was wrong.

I ran the controls. First result was uncomfortable.

On my balanced synthetic set, with the model predicting the SAME walls it received as input:

RAW       96.8%
DOMAIN    96.3%
BRYLA    100.0%
SHUFFLED  99.9%
RANDOM    96.2%

BRYLA beats RAW, DOMAIN and RANDOM — good. But BRYLA ≈ SHUFFLED. That was the important signal: shuffling which value belongs to which wall barely hurt. So the structure (the wall assignment) wasn’t being used — the model was essentially copying the value set from input to output. The test was measuring reconstruction, not meaning. Exactly the “predicting easy tags” trap John warned about.

Diagnosis: the values were self-identifying

In that setup each value was unique to one wall (“formal” only ever in REG, “negative” only in POL). So the model never needed the wall labels — the value itself revealed its wall. A bag of values was enough. Shuffling labels didn’t change the bag, so it didn’t hurt.

Redesigned test: hidden target + overlapping values

Two changes:

1. The model predicts a HIDDEN wall that is NOT in the input — so it must infer, not copy.
2. All walls share the same value space (A/B/C), so a value no longer reveals its wall. Now the model MUST use the assignment.

Target rule: cel = (S1 == S2) — requires distinguishing S1 from S2. Text carries only a neutral topic, never the target.

Result (2700 pairs, balanced, with and without text — nearly identical):

          chance   RAW    BRYLA   SHUFFLED   RANDOM
w/ text   66.7%   68.6%   100%    79.0%      66.4%
no text   66.7%   68.6%   100%    81.0%      68.6%

Now SHUFFLED collapses ~20 points below BRYLA. RANDOM sits at chance. RAW sits at chance. So: the gain comes from real values in their correct wall positions — not from prefix format, not from text.

The actual conclusion (conditional, not triumphant)

Putting both tests together:

unique values per wall   -> SHUFFLED = BRYLA  (structure redundant)
shared values per wall   -> SHUFFLED < BRYLA  (structure necessary)

So wall structure carries information exactly when values are NOT self- identifying. And overlapping values are precisely the natural-language case — “high” can be certainty, urgency, or intensity; “not” can be negation or something else. My first synthetic set was simply too clean, which hid the value of structure. That’s the honest result.

On the signature-vector idea

I also added a variant where the walls enter as ONE projected feature vector instead of ~28 tokens (the “fewer floats, no token tax” direction). On the clean set it matched the token form. I want to re-run it on the overlapping-values task before claiming anything — that’s the harder case and the fair test. Will report back.

What I’m NOT claiming

Still synthetic. This shows structure CAN matter and WHEN. It does not yet show that my parser, on real Polish text, produces enough value-overlap for structure to help in practice. That’s the next step you both pointed at: small natural dataset, clean target-only loss, parser coverage reported. Now I have a reason to invest in it — I know structure isn’t redundant a priori.

Thanks again. This thread moved the project further than the previous month did.