External Publication

[Research] From Functional Geometry to Dynamic Grammar: New LIMEN Audits (V23–V24) Across 7 Architectures

Hugging Face Forums [Unofficial] July 2, 2026

I broadened the search terms and found a few leads that may be useful:

I would search less for “symbolic dynamics of hidden states” alone and more for terms like activation-derived state abstraction , coarse state-transition structure , Markov/HMM-style hidden-state dynamics , internal state-transition dynamics during generation , cross-layer hidden-state dynamics , and belief-state geometry.

My impression is that your V24 framing is adjacent to state-transition abstractions over activation trajectories , rather than ordinary probing alone. I would not treat the papers below as identical to LIMEN, but they seem like useful comparison points and search anchors.

1. Nearby literature / comparison points

Area	Why it seems relevant
Markov/HMM-style activation-state abstraction	Closest to the idea of recovering coarse internal states and transition motifs from activations.
Hallucination detection via state transitions	Relevant to the B→D / grammar-violation idea.
Cross-layer hidden-state dynamics	Relevant because the signal is not just a static hidden vector, but how representations evolve across layers.
Belief-state geometry	A useful theoretical bridge from continuous representation geometry to coarse state dynamics.
Latent prediction trajectories	Useful for checking whether a “D” state is actually decision-like / commitment-like.

Closest matches I found

Markovian Circuit Tracing for Transformer State Dynamic This seems like the closest comparison point I found. It frames transformer computation as movement through internal states and tests whether activations contain coarse state-transition structure. The setup is synthetic HMM tasks where latent states, transition matrices, Bayesian belief vectors, Bayes-optimal predictions, and counterfactual targets are known.

I would not say this is the same setting as your work, because it is a controlled HMM benchmark rather than natural-language ambiguity across open-weight LLMs. But as a reference framework for activation-derived state abstractions + transition matrices + state forcing / centroid patching , it seems highly relevant.

PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics This seems especially relevant to your question about whether a B→D-like jump, or a violation of a B→A→D grammar, could be related to hallucination or reasoning failure. PoLLMgraph treats hallucination detection/forecasting as a problem of internal state-transition dynamics during generation.
ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs and its code repository This is not a symbolic-transition-grammar paper, but it is relevant because it shifts from static hidden-state probes to the cross-layer evolution of hidden states. That seems adjacent to your layer-wise dynamic-regime framing.
Transformers Represent Belief State Geometry in Their Residual Stream This seems like a useful theoretical bridge. It argues that, in HMM-like generative settings, transformer residual streams can linearly represent belief-state geometry. That does not directly prove anything about natural-language hallucination or B/A/D regimes, but it gives a vocabulary for connecting functional geometry to coarse state dynamics.
Eliciting Latent Predictions from Transformers with the Tuned Lens This is not a state-transition grammar method, but it is useful for checking whether a state called “D” is actually decision-like. Tuned Lens decodes hidden states into vocabulary distributions layer by layer, so it can help compare state occupancy with prediction commitment.
LogitLens4LLMs / repo Potentially useful if you want a more implementation-oriented logit-lens style tool for modern HF models, including Qwen-family models.

More detailed relationship to your B/A/D framing (click for more details)

2. Search terms that may help future readers

A useful phrase might be:

activation-derived coarse-state transition model

That wording avoids over-committing to “symbolic grammar,” while still capturing the useful part of the idea: you are turning continuous activation trajectories into discrete-ish regimes and then studying transitions among those regimes.

Other search terms I would try:

activation-derived state abstraction
coarse state-transition structure
Markovian hidden-state dynamics
HMM-style activation dynamics
internal state-transition dynamics during generation
cross-layer hidden-state dynamics
belief-state geometry in residual streams
latent prediction trajectories
layer-wise prediction refinement
hidden-state hallucination detection
activation trajectory clustering
state-transition interpretability
fixed-layer time-transition analysis
hidden-state dynamics hallucination detection

3. One thing I would separate: clustering vs naming

If B/A/D are meant as functional names rather than just cluster labels, I would separate two steps:

State construction How are the states obtained? Clustering? Thresholding? A linear probe? A hand-built rule? A model-specific classifier?
State naming Why is a state called exploration, stabilization, decision, etc.? What observable behavior supports that name?

For example, if D = decision-like , I would expect some relation to prediction commitment, such as:

lower next-token entropy,
larger top-1 / top-2 margin,
higher agreement with final-layer predictions,
earlier/later commitment timing,
possibly a Tuned Lens / Logit Lens trajectory becoming more stable.

But if D can be predicted almost entirely from entropy or top-token margin, then “D” may still be useful, but the interpretation shifts toward confidence proxy rather than an independent functional state.

Possible naming checks for B/A/D (click for more details)

4. A small routing diagnostic I tried

I also tried a small Colab-T4-style routing check. I would not treat this as validation or refutation of the dynamic-grammar idea. The point was only to see what simple state abstractions tend to pick up.

The setup was intentionally lightweight:

small public models,
manual greedy decoding,
hidden states collected with output_hidden_states=True,
raw vs layer-centered vs layer-z-scored hidden states,
PCA/KMeans-style state abstraction,
checks against depth-only, prompt-family, entropy/margin, generation-step, shuffled-layer, and fixed-layer time-transition baselines.

The most useful observation was:

Raw layer-wise transition motifs can look clean while being strongly aligned with layer depth.

Layer centering or z-scoring can reduce that signal in some models, but it can also expose other proxy alignments such as prompt family, entropy/margin bins, or generation step. So I would frame these checks as routing diagnostics , not as an objection to the dynamic-grammar idea.

What the routing diagnostic suggested (click for more details)

5. Suggested reporting table

If you continue this line of work, I think a compact table like this would make the claim easier to compare across models and future replications.

View	What it tests	Useful controls
raw layer-wise states	Original observed motif	shuffled-layer, random Gaussian
layer-centered states	Non-depth residual structure	depth-only, prompt-family
layer-z-scored states	Mean/scale controlled structure	entropy/margin, step
fixed-layer time transitions	Generation-time dynamics without layer-depth transitions	shuffled-time, prompt-family
D-state checks	Whether “D” is decision-like	entropy, margin, tuned/logit lens
B→D violation checks	Whether transition skips predict failure	hallucination / reasoning-error cases
held-out prompt families	Generality beyond prompt template	prompt-family split
held-out model families	Generality beyond one architecture	model-family split

A minimal version could be:

raw / layer-centered / layer-z-scored
×
depth / prompt-family / entropy / margin / step / shuffled-order / fixed-layer time-transition

That would make the B/A/D interpretation much easier to evaluate without requiring a huge model run.

6. Reproducibility note: hidden-state extraction contract

Not as an objection to your result, but for this kind of geometry/transition analysis, I think the exact hidden-state extraction contract matters a lot.

The HF docs note that outputs.hidden_states[-1] may not always match outputs.last_hidden_state, because some models apply normalization or other processing to the returned last hidden state: Transformers model output docs.

Generation-time hidden states also need care. There is an HF Forum thread on retrieving hidden states from model.generate() where the answer separates generated-token states from input-sequence states and mentions return_dict_in_generate=True / output_hidden_states=True: The hidden_states when I use model.generate.

There is also a Transformers issue where model() and model.generate() hidden states did not match as expected in a Qwen2.5 example: Hidden states are different for model() and model.generate(). I would not read that issue as applying directly to your setup, but it is a good reminder that extraction details should be explicit.

Extraction details I would report (click for more details)

7. Probe / format confound caution

One related caution is that high hidden-state separability does not automatically imply a functional internal variable. A recent example is Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States, where high probe accuracy on Qwen3-14B was explained by format confounds such as source identity, option count, and response length.

I would not treat that as a refutation of your result. I would treat it as a useful reminder to include format/depth/confidence controls before assigning strong functional interpretations to state labels.

In this context, the main question becomes:

Are B/A/D labels still informative after controlling for layer depth, prompt family, response format, token position, entropy, top-token margin, and generation step?

If yes, the dynamic-grammar interpretation becomes easier to compare. If no, the result may still be useful, but the interpretation shifts toward depth/confidence/prompt-format dynamics.

8. Possible next tracks

I would split the next steps into three tracks.

Track A — terminology / literature mapping

Use these as comparison anchors:

Markovian Circuit Tracing for activation-state abstraction and transition matrices.
PoLLMgraph for hallucination as internal state-transition dynamics.
ICR Probe for cross-layer hidden-state evolution.
Belief State Geometry for the continuous-geometry side.
Tuned Lens / LogitLens4LLMs for prediction-trajectory checks.

Track B — observational controls

Before moving to large models or causal claims, I would add a small diagnostic table:

raw vs layer-centered vs layer-z-scored states,
depth-only baseline,
entropy/margin-only baseline,
prompt-family baseline,
generation-step baseline,
shuffled-layer transition matrix,
random Gaussian baseline,
fixed-layer time-transition matrix.

This is relatively cheap and makes the interpretation much clearer.

Track C — later predictive / causal validation

Once the observational state definition is stable:

test whether B→D-like jumps predict factual hallucination or reasoning errors,
compare correct vs incorrect generations,
try held-out prompt families,
try model-family-held-out mapping,
eventually test centroid patching / activation patching.

For tooling, TransformerLens is useful for caching and editing activations in supported models, and nnsight has activation-patching style workflows. I would treat that as a later stage, after the observational labels are stable.

9. How I would frame the claim

I would avoid framing the current result as either:

“a universal transformer grammar,” or
“just a probing artifact.”

A safer and more useful middle formulation might be:

Under this extraction and labeling setup, there appear to be recurrent activation-derived transition motifs. The next step is to report which motifs survive depth, confidence, prompt-family, step, shuffled-order, and fixed-layer controls.

That keeps the core idea alive while making it easier for other people to reproduce, compare, and extend.

In short: the B/A/D grammar idea seems worth pursuing, but I would make the next iteration less about proving the names and more about making the state abstraction auditable.