Exploring Functional Regimes Inside Small Language Models (Independent Research)

Hi everyone,

Over the last few months I’ve been working on an independent research project exploring the internal dynamics of small and medium-sized language models.

Rather than evaluating models only by their outputs (benchmarks, perplexity, etc.), I’m trying to characterize how their hidden representations evolve during inference.

The project currently covers 7 open models:

• GPT-2
• DistilGPT2
• OPT-125M
• Qwen2.5-0.5B-Instruct
• TinyLlama-1.1B-Chat
• Phi-1.5
• Llama-3.2-1B

Phase 1 — Dynamic behavior

The first part of the framework studies internal trajectories through hidden-state dynamics.

Instead of asking “Which model is more accurate?” , I ask:

• How stable are hidden-state trajectories?
• When do bifurcations appear?
• How does commitment evolve?
• Can different architectures be grouped according to their internal dynamics?

This produced several reproducible dynamical fingerprints and architecture clusters.

Phase 2 — Functional organization

The second phase moves away from pure dynamics and investigates whether different functional properties become linearly decodable inside hidden representations.

Across multiple probe experiments I observed evidence that:

• functional information is linearly decodable,
• several functional properties can coexist within the same layer,
• some layers have substantially higher functional capacity than others.

One interesting observation is that the position of these high-capacity regions varies across architectures rather than appearing at identical absolute depths.

Phase 3 — Geometry

The result that surprised me the most came from a series of control experiments.

After training linear probes I compared:

• Gaussian perturbations
• Feature permutation
• Orthogonal rotations

Gaussian noise and feature permutation substantially reduced decodability.

Orthogonal rotations, however, preserved it almost entirely.

That suggests (at least empirically) that the functional signal depends more on the geometry of the representation space than on specific embedding dimensions.

This seems broadly consistent with ideas discussed in mechanistic interpretability about distributed feature directions.

Architecture Clusters

Across several independent audits, the models repeatedly separate into two broad behavioral groups.

Cluster A

• GPT-2
• DistilGPT2

These models consistently exhibit similar dynamic and functional profiles.

Cluster B

• OPT-125M
• Qwen2.5-0.5B-Instruct
• TinyLlama-1.1B-Chat
• Phi-1.5
• Llama-3.2-1B

Despite architectural differences, these models repeatedly cluster together across multiple analyses.

Seeing the same grouping emerge from different metrics was one of the motivations for continuing the project.

Current Direction

I’m now moving from observation toward causal testing.

The next experiments aim to answer questions such as:

• Can perturbing specific functional regions causally alter downstream behavior?
• How much functional information can a single hidden representation simultaneously support?
• Are these organizational principles preserved as model size increases?

This is entirely independent research, so I’d genuinely appreciate feedback.

I’m especially interested in hearing from people working on:

• mechanistic interpretability,
• representation geometry,
• linear probes,
• sparse autoencoders,
• causal interventions,
• hidden-state analysis.

I’d love to know whether these observations resonate with existing work—or whether there are obvious control experiments I should run next.