{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiekgfi6hembi5unyt72he6lw355wab4qpyzajdv6mnpw52uk6eysq",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mpfj3fyedzs2"
  },
  "path": "/t/exploring-functional-regimes-inside-small-language-models-independent-research/177219#post_1",
  "publishedAt": "2026-06-29T02:23:53.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "Hi everyone,\n\nOver the last few months I’ve been working on an independent research project exploring the **internal dynamics** of small and medium-sized language models.\n\nRather than evaluating models only by their outputs (benchmarks, perplexity, etc.), I’m trying to characterize **how their hidden representations evolve during inference**.\n\nThe project currently covers 7 open models:\n\n  * GPT-2\n  * DistilGPT2\n  * OPT-125M\n  * Qwen2.5-0.5B-Instruct\n  * TinyLlama-1.1B-Chat\n  * Phi-1.5\n  * Llama-3.2-1B\n\n\n\n* * *\n\n## Phase 1 — Dynamic behavior\n\nThe first part of the framework studies internal trajectories through hidden-state dynamics.\n\nInstead of asking _“Which model is more accurate?”_ , I ask:\n\n  * How stable are hidden-state trajectories?\n  * When do bifurcations appear?\n  * How does commitment evolve?\n  * Can different architectures be grouped according to their internal dynamics?\n\n\n\nThis produced several reproducible dynamical fingerprints and architecture clusters.\n\n* * *\n\n## Phase 2 — Functional organization\n\nThe second phase moves away from pure dynamics and investigates whether different functional properties become linearly decodable inside hidden representations.\n\nAcross multiple probe experiments I observed evidence that:\n\n  * functional information is linearly decodable,\n  * several functional properties can coexist within the same layer,\n  * some layers have substantially higher functional capacity than others.\n\n\n\nOne interesting observation is that the position of these high-capacity regions varies across architectures rather than appearing at identical absolute depths.\n\n* * *\n\n## Phase 3 — Geometry\n\nThe result that surprised me the most came from a series of control experiments.\n\nAfter training linear probes I compared:\n\n  * Gaussian perturbations\n  * Feature permutation\n  * Orthogonal rotations\n\n\n\nGaussian noise and feature permutation substantially reduced decodability.\n\nOrthogonal rotations, however, preserved it almost entirely.\n\nThat suggests (at least empirically) that the functional signal depends more on the **geometry of the representation space** than on specific embedding dimensions.\n\nThis seems broadly consistent with ideas discussed in mechanistic interpretability about distributed feature directions.\n\n* * *\n\n## Architecture Clusters\n\nAcross several independent audits, the models repeatedly separate into two broad behavioral groups.\n\n**Cluster A**\n\n  * GPT-2\n  * DistilGPT2\n\n\n\nThese models consistently exhibit similar dynamic and functional profiles.\n\n**Cluster B**\n\n  * OPT-125M\n  * Qwen2.5-0.5B-Instruct\n  * TinyLlama-1.1B-Chat\n  * Phi-1.5\n  * Llama-3.2-1B\n\n\n\nDespite architectural differences, these models repeatedly cluster together across multiple analyses.\n\nSeeing the same grouping emerge from different metrics was one of the motivations for continuing the project.\n\n* * *\n\n## Current Direction\n\nI’m now moving from observation toward causal testing.\n\nThe next experiments aim to answer questions such as:\n\n  * Can perturbing specific functional regions causally alter downstream behavior?\n  * How much functional information can a single hidden representation simultaneously support?\n  * Are these organizational principles preserved as model size increases?\n\n\n\n* * *\n\nThis is entirely independent research, so I’d genuinely appreciate feedback.\n\nI’m especially interested in hearing from people working on:\n\n  * mechanistic interpretability,\n  * representation geometry,\n  * linear probes,\n  * sparse autoencoders,\n  * causal interventions,\n  * hidden-state analysis.\n\n\n\nI’d love to know whether these observations resonate with existing work—or whether there are obvious control experiments I should run next.",
  "title": "Exploring Functional Regimes Inside Small Language Models (Independent Research)"
}