{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiaujddif5u7rmtspu32aryjik56bvams7a5n3r4p4zsu2fb2vsuvm",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mokfhcb76wl2"
  },
  "path": "/t/introducing-drm-language-emitter/176947#post_1",
  "publishedAt": "2026-06-18T07:02:51.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "I’m sharing an experimental research project called **DRM Language Emitter**.\n\nIt is a geometry-first language model lab for exploring generative AI without Transformer blocks, without self-attention, without Q/K/V attention, and without KV cache inside the DRM model. The central idea is to treat language generation not as attention over a token window, but as controlled motion through a learned relational manifold.\n\nIn DRM, generation follows a different path:\n\n\n    token\n    -> latent state\n    -> active directions\n    -> learned relational metric\n    -> controlled latent motion\n    -> next latent state\n    -> token logits\n\n\nThe working hypothesis is simple:\n\n> Maybe language can be generated as motion through a learned relational state space.\n\nThis is not a production model.\n\nThis is not a claim that DRM is better than Transformers in general.\n\nThis is not a claim that DRM is better than world models in general.\n\nIt is a research scaffold for testing whether explicit geometry, active directions, latent dynamics, and learned metrics can become useful components for small language models and symbolic dynamics. The README describes the model’s central computation as a latent trajectory through state, directions, gates, metric, velocity, state update, and logits.\n\n## Why I built this\n\nMost current language modeling research is organized around the Transformer paradigm.\n\nThat makes sense. Transformers work extremely well.\n\nBut I wanted to test a different question:\n\n> What happens if the model does not attend backward over a context window, but instead carries an evolving latent state through a learned geometry?\n\nDRM Language Emitter is my attempt to explore that question in code.\n\nThe model has:\n\n  * a recurrent latent state;\n  * learned active directions;\n  * gates and effective active dimension;\n  * a relational metric of the form `diag + U U^T`;\n  * controlled latent motion;\n  * a language emitter that decodes the updated state into token logits.\n\n\n\nThe model is autoregressive, but its memory is the evolving latent state rather than attention over a token sequence.\n\n## What makes it different\n\nDRM Language Emitter does not use:\n\n  * Transformer blocks;\n  * self-attention;\n  * Q/K/V attention;\n  * `nn.MultiheadAttention`;\n  * KV cache.\n\n\n\nInstead, it tries to make the geometry of generation explicit and measurable.\n\nThe project logs diagnostics such as cross-entropy, approximate perplexity, metric action, active dimension, gate entropy, metric norm, condition proxy, recurrence, stability, low-action path diagnostics, and symbolic world-modeling metrics.\n\nThat matters because I do not want the model to be only a black box that outputs tokens.\n\nI want to inspect how it moves.\n\nI want to measure whether the learned geometry collapses, stabilizes, expands, or forms useful trajectories.\n\n## What this repo contains\n\nThe repository includes:\n\n\n    src/drm_language_emitter/   DRM model package\n    transformer/                tiny Transformer baseline\n    world_model/                tiny symbolic world-model baseline\n    scripts/                    training, generation, evaluation, sweeps, dashboards\n    configs/                    DRM and benchmark configs\n    docs/                       math, limitations, competition notes, benchmark artifacts\n    tests/                      smoke and invariant tests\n\n\nIt is CPU-runnable, with CUDA optional. The latest local benchmark reported CPU-only execution, so stronger CUDA and time-matched comparisons are still future work.\n\n## What I am trying to learn\n\nThe long-term research question is:\n\n> Can learned geometry become a useful primitive for language generation?\n\nMore specifically:\n\n  * Can active directions replace attention-like routing in tiny regimes?\n  * Can a learned relational metric help structure token generation?\n  * Can latent motion provide memory without KV cache?\n  * Can this architecture model symbolic worlds serialized as language?\n  * Can it compete with tiny Transformers under controlled conditions?\n  * Can it become a discrete dynamics/emission module for text-native world models?\n\n\n\nI do not know the final answer yet.\n\nThat is why the repo exists.\n\n## Quick start\n\nInstall:\n\n\n    pip install -e .\n\n\nTrain a tiny DRM model:\n\n\n    python scripts/train_tiny.py --config configs/tiny.yaml --text data/tiny.txt\n\n\nGenerate text:\n\n\n    python scripts/generate.py --checkpoint runs/tiny/drm_tiny.pt --prompt \"DRM \"\n\n\nRun geometry diagnostics:\n\n\n    python scripts/eval_geometry.py --checkpoint runs/tiny/drm_tiny.pt\n    python scripts/eval_geodesic_paths.py --checkpoint runs/tiny/drm_tiny.pt\n\n\n## Early benchmark signal\n\nAt the end of the current README, I also include benchmark artifacts comparing:\n\n  * DRM Language Emitter\n  * Tiny Transformer\n  * Tiny Symbolic World Model\n\n\n\nThe latest tiny symbolic world-model benchmark used a deterministic gridworld serialized as text. It produced:\n\n\n    runs: 72\n    aggregate rows: 24\n\n\nThe top result by next-state exact match was:\n\n\n    drm_tiny @ 2000 steps\n    next_state_exact_match = 0.0751\n\n\nIn the same benchmark, `transformer_tiny_220k @ 3000` had a lower invalid-state rate of `0.0026`, and the tiny supervised world model reached low CE but weak exact-match and rollout metrics.\n\nThe honest interpretation is:\n\n> DRM showed an early signal on symbolic next-state prediction, but the absolute accuracy is still low. This is diagnostic, not decisive.\n\nThe benchmark does not prove that DRM is better than Transformers.\n\nIt does not prove that DRM is better than world models.\n\nIt does not say anything about large multimodal world models.\n\nIt only shows that this geometry-first emitter is now testable against baselines in a controlled tiny symbolic environment.\n\n## Current scientific status\n\nAllowed claim:\n\n> DRM Language Emitter is a functional non-Transformer language model prototype with explicit, measurable geometry and controlled tiny comparisons against Transformer and symbolic world-model baselines.\n\nNot allowed:\n\n> DRM is broadly better than Transformers or world models.\n\nThat distinction matters.\n\n## Next steps\n\nI am currently working on:\n\n  * constrained symbolic decoding;\n  * stronger rollout evaluation;\n  * CUDA and time-matched runs;\n  * more baselines;\n  * curriculum variants for symbolic worlds;\n  * ablations around metric, gates, and active dimension;\n  * testing whether the learned geometry matters beyond cross-entropy.\n\n\n\nRepository:\n\n\n    https://github.com/gnai-creator/drm-language-emitter\n\n\nFeedback, criticism, reproduction attempts, and benchmark suggestions are very welcome.",
  "title": "Introducing DRM Language Emitter"
}