Custom semantic representation ("bryła") beats raw text in 24/27 configs — built solo on an RTX 2060, looking for feedback

Hm. If we also merge in KnackAU’s approach, maybe the next step could be something like this:

---

This looks like the sharpest version of the experiment so far.

The important part is not just that BRYLA won. The important part is the sequence:

1. the first control result was uncomfortable;
2. BRYLA ≈ SHUFFLED showed that the test was not really measuring structure;
3. the reason was diagnosed: values were self-identifying;
4. the task was redesigned with overlapping values;
5. after that, BRYLA separated clearly from SHUFFLED, while RAW and RANDOM stayed near chance.

That is a much stronger result than the earlier “Bryła beats RAW” story.

1. Why the redesigned test is much better

The first version had a hidden flaw:

unique values per wall -> wall labels become redundant

If formal only ever belongs to REG, and negative only ever belongs to POL, then the model does not need the structure. It only needs the set of values.

So this result:

BRYLA    100.0%
SHUFFLED  99.9%

was actually the key diagnostic signal. It showed that the model could solve the task from value identity alone.

The redesigned version is much better:

all walls share A/B/C
target = hidden wall
cel = (S1 == S2)

Now the model must know which value belongs to which wall. A bag of values is no longer enough.

That makes the new result meaningful:

RAW/RANDOM ≈ chance
SHUFFLED   ≈ 79–81%
BRYLA      = 100%

My interpretation:

The new test finally creates a condition where wall assignment matters. That is the right kind of synthetic test for Bryła.

2. How I would state the current conclusion

I would still keep the claim conditional, but it is now much stronger:

In the controlled overlapping-values setup, Bryła is not merely adding prefix format, domain metadata, or copyable values. It carries useful information through correct field/value assignment.

But I would not yet claim:

Bryła generally beats raw text on natural Polish data.

The current result shows:

structure can matter
and
when structure matters

The next step is to test whether natural Polish data has enough of this kind of ambiguity.

Natural language probably does contain many overlapping-value cases:

Value	Possible meanings depending on field
high	urgency, certainty, intensity, importance
negative	sentiment, polarity, negation, evaluation
reported	source type, evidentiality, discourse status
formal	register, document type, communication style
core	salience, topic focus, answer relevance

So the redesigned synthetic test is not arbitrary. It is a simplified version of a real problem.

3. Where KnackAU’s suggestion fits

I think KnackAU’s “fewer floats, no token tax” idea should not be treated as a competing direction.

It is better seen as another carrier for the same structure:

Bryła structure -> token serialization -> model
Bryła structure -> projected vector / conditioning signal -> model

So the core question becomes:

Is Bryła useful because it is a token sequence, or because it is structured side information?

Those are different hypotheses.

The token form has advantages:

Token Bryła	Pros	Cons
[S1:A] [S2:B] ...	readable, debuggable, easy to ablate	token tax, possible copying shortcuts, sequence-order sensitivity

The vector form has different advantages:

Vector Bryła	Pros	Cons
projected signature vector	no token tax, compact, closer to latent conditioning	less interpretable, implementation details matter, may collapse assignments if designed poorly

This connects to several existing ideas:

• FiLM: feature-wise linear modulation; conditioning information changes intermediate features.
• Feature-wise transformations — Distill: clear explanation of how conditioning can be injected into neural networks.
• Prefix-Tuning: continuous task-specific vectors act like virtual prefix tokens.
• Prompt Tuning: soft prompts condition models without discrete text prompts.
• MeCo: metadata conditioning during training, followed by cooldown without metadata.

These are not identical to Bryła, but they support the general idea that side information can be passed either as tokens or as continuous conditioning.

4. Suggested next experiment: token Bryła vs vector Bryła

I would run the next comparison on the exact redesigned overlapping-values task, not on an easier clean set.

Suggested conditions:

RAW
BRYLA-token
BRYLA-vector
BRYLA-token+vector
SHUFFLED-token
SHUFFLED-vector
RANDOM-vector

The reason to use the overlapping-values task is simple:

if the value itself identifies the wall, the test is too easy
if values overlap across walls, assignment becomes necessary

5. How to interpret the outcomes

Result	Interpretation
BRYLA-vector ≈ BRYLA-token	vector form may preserve the useful structure while avoiding token tax
BRYLA-vector > BRYLA-token	structure may be better injected as latent conditioning than as text
BRYLA-token > BRYLA-vector	small models may learn explicit symbolic tokens more easily
BRYLA-token+vector > both	token and vector channels may be complementary
SHUFFLED-vector high	vector representation may be preserving value set but losing assignment
RANDOM-vector near chance	good; vector channel is not just a generic regularizer
RAW near chance	good; target is not recoverable from text alone

The most important comparison is:

BRYLA-vector vs SHUFFLED-vector

If vector Bryła beats shuffled-vector on the overlapping-values task, then the vector really preserves assignment structure.

6. Implementation variants worth separating

“Vector Bryła” can mean several different things. I would separate them if possible:

Variant	Description	Why test it
projected global vector	encode all fields into one vector	closest to “fewer floats”
BOS addition	add vector to first token embedding	simple implementation
prefix virtual token	convert vector into one or more learned prefix embeddings	close to prefix-tuning
FiLM-style modulation	use vector to scale/shift hidden features	stronger conditioning mechanism
token + vector	keep readable tokens and add vector side channel	tests complementarity

A minimal first version could be:

BRYLA-token
BRYLA-one-vector
BRYLA-token+one-vector
SHUFFLED-one-vector
RANDOM-one-vector

That is already enough to answer the main question.

7. Metrics I would report

For the synthetic overlapping-values test:

Metric	Why
accuracy	main task score
chance baseline	sanity check
token count	shows token-tax advantage
train time	important on small hardware
inference time	practical cost
seeds	stability
shuffled gap	measures structure sensitivity

For natural data later:

Metric	Why
clean target-only PPL	avoids easy-tag artifact
task metric / F1 / chrF	checks actual output quality
parser OTHER%	measures parser coverage
field entropy	detects dead/default fields
value-overlap ratio	measures whether structure is actually needed
token cost	practical tradeoff

8. Bridge to natural data

After token-vs-vector is tested on the overlapping-values setup, I would move to a small natural dataset.

Not huge. Something like:

6 domains × 50 examples = 300 examples

Suggested domains:

technical
geography
biography
science
daily life
sports/events

Run:

RAW
DOMAIN
BRYLA-token
BRYLA-vector
SHUFFLED-BRYLA
RANDOM-BRYLA

The key question:

Does Bryła beat DOMAIN and SHUFFLED on natural data?

If yes, the result becomes much stronger.

If no, it is still informative:

Outcome	Meaning
DOMAIN ≈ BRYLA	Bryła may mostly encode domain/topic
SHUFFLED ≈ BRYLA	field assignment may not matter enough in that data
VECTOR ≈ TOKEN	vector may remove token tax
TOKEN > VECTOR	explicit symbolic representation may be easier for small models
VECTOR > TOKEN	latent conditioning may be the better interface

9. One extra diagnostic: value-overlap ratio

The most useful new diagnostic from this update is probably:

value-overlap ratio

Rough idea:

How often can the same value occur in multiple fields/walls?

If values are field-unique, structure is less necessary.

If values overlap across fields, assignment matters.

So I would report:

unique values per field
shared values across fields
field entropy
default ratio
assignment dependency

This directly connects the synthetic result to natural language.

10. Suggested short conclusion

My current read:

The redesigned overlapping-values task is a real improvement because it removes the self-identifying-value shortcut. KnackAU’s vector idea is also worth integrating, because it tests whether Bryła’s useful part is the token serialization or the underlying structured signal. The next best experiment is probably token Bryła vs projected-vector Bryła on the same overlapping-values task, with shuffled-vector and random-vector controls.

If that works, then the natural next stage is:

small natural Polish dataset
+ DOMAIN control
+ SHUFFLED control
+ RANDOM control
+ token vs vector Bryła
+ clean target-only metrics
+ parser coverage

That would make the result much harder to dismiss.