{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreifabl7veuhcvpeehzkrnmbitivua4nea4pobmtci5yh4w5jry5xou",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mokd65tczsy2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreifvmv2x2lmuxfufoy5i5zkd6qie2zszmah2ha6kenz37znrf6cc7e"
    },
    "mimeType": "image/webp",
    "size": 74274
  },
  "path": "/felipe_muniz_grsba/introducing-drm-language-emitter-language-generation-as-motion-through-learned-geometry-3a5l",
  "publishedAt": "2026-06-18T07:33:44.000Z",
  "site": "https://dev.to",
  "tags": [
    "ai",
    "opensource",
    "machinelearning",
    "https://github.com/gnai-creator/drm-language-emitter"
  ],
  "textContent": "Most language models today are built around the Transformer paradigm.\n\nThat makes sense.\n\nTransformers work.\n\nThey scale.\n\nThey dominate modern NLP.\n\nBut I wanted to explore a different question:\n\n> What if language generation does not need to be modeled as attention over a context window?\n\nWhat if a model could generate language by carrying an evolving latent state through a learned geometry?\n\nThat is the idea behind **DRM Language Emitter**.\n\nRepository:\n\nhttps://github.com/gnai-creator/drm-language-emitter\n\n##  What is DRM Language Emitter?\n\n**DRM Language Emitter** is an experimental, geometry-first language model lab.\n\nIt is not a Transformer.\n\nInside the DRM model, it does not use:\n\n  * Transformer blocks\n  * self-attention\n  * Q/K/V attention\n  * `nn.MultiheadAttention`\n  * KV cache\n\n\n\nInstead, it treats language generation as controlled motion through a learned relational manifold.\n\nThe basic flow is:\n\n\n\n    token\n      -> latent state z_t\n      -> active directions\n      -> learned relational metric\n      -> controlled latent motion\n      -> next latent state z_{t+1}\n      -> token logits\n\n\nThe model is still autoregressive.\n\nBut its memory is not attention over a token sequence.\n\nIts memory is the evolving latent state.\n\n##  The core hypothesis\n\nThe working hypothesis is:\n\n> Language generation can be modeled as motion through a learned relational state space.\n\nThat means the model does not simply ask:\n\n\n\n    Which previous tokens should I attend to?\n\n\nIt asks something closer to:\n\n\n\n    Where am I in latent space?\n    Which directions are active?\n    How expensive is movement under the learned metric?\n    How should the state move before emitting the next token?\n\n\nThis is why I call it a geometry-first language emitter.\n\n##  A simplified architecture\n\nThe architecture can be summarized as:\n\n\n\n    input_ids\n       |\n    TokenEmbedding\n       |\n    for each time step:\n       |\n       z_t\n       |\n    DirectionField(z_t)\n       -> directions V(z_t)\n       -> gates a(z_t)\n       -> effective active dimension dimD\n       |\n    RelationalMetric(z_t)\n       -> diag + U U^T\n       |\n    DRMFlow(z_t, token_embedding, directions, gates)\n       -> dz\n       |\n    Metric action g_z(dz, dz)\n       |\n    StateUpdater\n       -> z_{t+1}\n       |\n    LanguageEmitter(z_{t+1})\n       -> logits\n\n\nA minimal conceptual version looks like this:\n\n\n\n    for token in sequence:\n        embedding = token_embedding(token)\n\n        directions, gates = direction_field(z)\n        metric = relational_metric(z)\n\n        dz = drm_flow(z, embedding, directions, gates)\n        action = metric_action(metric, dz)\n\n        z = state_updater(z, dz)\n        logits = language_emitter(z)\n\n\nThe important part is that the model has an explicit internal geometry.\n\nIt can log and measure:\n\n  * metric action\n  * active dimension\n  * gate entropy\n  * metric norm\n  * condition proxy\n  * recurrence\n  * stability\n  * low-action path diagnostics\n\n\n\nThis makes the model interesting not only as a generator, but also as an object of study.\n\n##  Why not just use a Transformer?\n\nA Transformer is the correct baseline.\n\nThat is why the repository includes tiny Transformer comparisons.\n\nBut the goal of DRM is not to replace Transformers by declaration.\n\nThe goal is to test whether a different computational primitive can be useful in small regimes.\n\nThe Transformer primitive is attention.\n\nThe DRM primitive is controlled latent motion under a learned metric.\n\nThese are very different assumptions.\n\nA Transformer builds context by looking backward.\n\nDRM carries context by evolving state forward.\n\nA Transformer computes token-token interactions.\n\nDRM computes state-motion-emission dynamics.\n\n##  Why geometry?\n\nBecause geometry gives us measurable structure.\n\nIf language is treated as a trajectory, we can ask questions like:\n\n  * Does the latent state collapse?\n  * Does the metric become unstable?\n  * Which directions are active?\n  * Is the model moving through a narrow or broad region?\n  * Is generation smooth or chaotic?\n  * Do symbolic transitions correspond to stable latent movement?\n\n\n\nThis opens the door to diagnostics that are harder to express in a standard black-box token predictor.\n\nThe goal is not mystical geometry.\n\nThe goal is measurable geometry.\n\n##  Repository structure\n\nThe repository contains:\n\n\n\n    src/drm_language_emitter/   DRM model package\n    transformer/                tiny Transformer baseline\n    world_model/                tiny symbolic world-model baseline\n    scripts/                    training, generation, evaluation, sweeps, dashboards\n    configs/                    DRM and benchmark configs\n    docs/                       math, limitations, competition notes, benchmark artifacts\n    tests/                      smoke and invariant tests\n\n\nThe project is CPU-runnable.\n\nCUDA is optional.\n\n##  Quick start\n\nInstall:\n\n\n\n    pip install -e .\n\n\nTrain a tiny DRM model:\n\n\n\n    python scripts/train_tiny.py \\\n      --config configs/tiny.yaml \\\n      --text data/tiny.txt\n\n\nGenerate text:\n\n\n\n    python scripts/generate.py \\\n      --checkpoint runs/tiny/drm_tiny.pt \\\n      --prompt \"DRM \"\n\n\nRun geometry diagnostics:\n\n\n\n    python scripts/eval_geometry.py \\\n      --checkpoint runs/tiny/drm_tiny.pt\n\n    python scripts/eval_geodesic_paths.py \\\n      --checkpoint runs/tiny/drm_tiny.pt\n\n\n##  Tiny benchmark: DRM vs Transformer vs World Model\n\nThe repository also includes a small symbolic benchmark.\n\nThis benchmark compares:\n\n  * DRM Language Emitter\n  * Tiny Transformer\n  * Tiny supervised symbolic world model\n\n\n\nThe task is a deterministic symbolic gridworld serialized as text.\n\nThe models need to predict symbolic transitions such as:\n\n\n\n    state + action -> next state + reward + done\n\n\nThis is not visual world modeling.\n\nThis is not a benchmark against large multimodal world models.\n\nIt is a tiny symbolic text-world designed to test whether models can learn discrete dynamics expressed as language.\n\n##  Metrics\n\nThe benchmark reports:\n\n  * validation cross-entropy\n  * next-state exact match\n  * rollout exact match\n  * reward accuracy\n  * done accuracy\n  * invalid state rate\n  * parameter count\n  * elapsed time\n  * tokens seen\n  * throughput\n\n\n\nThis is important because low loss alone does not necessarily mean correct symbolic dynamics.\n\nA model can learn token-level regularities while still failing to predict exact state transitions.\n\n##  Latest local result\n\nThe completed benchmark produced:\n\n\n\n    runs: 72\n    aggregate rows: 24\n\n\nTop results by next-state exact match:\n\nModel | Steps | Family | Next-state exact match | Rollout exact match | Best CE | Invalid state rate | Params\n---|---|---|---|---|---|---|---\n`drm_tiny` | 2000 | DRM | 0.0751 | 0.0058 | 0.5511 | 0.1328 | 92,710\n`transformer_tiny_220k` | 3000 | Transformer | 0.0563 | 0.0000 | 0.4008 | 0.0026 | 220,208\n`transformer_tiny_93k` | 2000 | Transformer | 0.0516 | 0.0000 | 0.4594 | 0.2969 | 93,872\n`world_model_tiny` | 2000 | World Model | 0.0476 | 0.0000 | 0.2573 | 0.4668 | 102,051\n`world_model_tiny` | 3000 | World Model | 0.0415 | 0.0000 | 0.2497 | 0.4668 | 102,051\n\nThe most interesting part is not that DRM “wins everything”.\n\nIt does not.\n\nThe result is more nuanced:\n\n  * DRM had the best next-state exact match in this tiny symbolic text-world.\n  * Transformer 220k had the lowest invalid-state rate.\n  * The tiny supervised world model reached low CE, but did not convert that into strong exact-match or rollout performance.\n  * Rollout exact match is still very low.\n\n\n\nSo the honest interpretation is:\n\n> DRM shows an early signal on symbolic next-state prediction, but the benchmark is still diagnostic, not decisive.\n\n##  The main lesson\n\nFor me, the most important takeaway is:\n\n> Low token-level cross-entropy does not automatically imply correct symbolic transition modeling.\n\nThat matters for world-model-like tasks.\n\nIf a model is supposed to represent dynamics, then we should not only ask whether it predicts likely tokens.\n\nWe should also ask whether it predicts valid states, exact transitions, and coherent rollouts.\n\n##  What I am not claiming\n\nI am not claiming that DRM is better than Transformers in general.\n\nI am not claiming that DRM is better than world models in general.\n\nI am not claiming that this benchmark says anything about large multimodal world models.\n\nI am not claiming robust long-horizon planning.\n\nThis is a small research scaffold.\n\nThe results are early.\n\nThe exact-match values are still low.\n\nThe model needs more work.\n\n##  What I am claiming\n\nDRM Language Emitter is a functional non-Transformer language model prototype.\n\nIt has explicit, measurable geometry.\n\nIt can be compared against Transformer and symbolic world-model baselines.\n\nAnd in a tiny symbolic text-world benchmark, it showed an interesting signal on next-state exact match.\n\nThat is enough to keep investigating.\n\n##  Reproducing the symbolic benchmark\n\nGenerate the dataset:\n\n\n\n    python scripts/make_tiny_world_dataset.py \\\n      --output-root data/tiny_world \\\n      --seed 1 \\\n      --grid-size 5 \\\n      --num-train 20000 \\\n      --num-val 2000 \\\n      --max-rollout-len 8\n\n\nRun the sweep:\n\n\n\n    python scripts/sweep_world_model_competition.py \\\n      --steps 1000 2000 3000 \\\n      --seeds 1 2 3 \\\n      --dataset-root data/tiny_world \\\n      --output-root runs/world_model_competition\n\n\nGenerate the dashboard:\n\n\n\n    python scripts/make_world_model_dashboard.py \\\n      --root runs/world_model_competition \\\n      --title \"DRM vs Transformer vs Tiny Symbolic World Model\"\n\n\n##  Next steps\n\nThe next things I want to improve are:\n\n  * constrained symbolic decoding\n  * stronger rollout evaluation\n  * CUDA and time-matched runs\n  * curriculum variants for symbolic worlds\n  * more baselines\n  * better ablations around metric, gates, and active dimension\n  * tests to isolate whether learned geometry helps beyond cross-entropy\n\n\n\n##  Final thought\n\nThis project started from a simple intuition:\n\nMaybe language generation can be treated as movement.\n\nNot metaphorically.\n\nComputationally.\n\nA token enters.\n\nA state moves.\n\nA geometry shapes the motion.\n\nA new token is emitted.\n\nThat is DRM Language Emitter.\n\nRepository:\n\nhttps://github.com/gnai-creator/drm-language-emitter\n\nFeedback, criticism, reproduction attempts, and benchmark suggestions are welcome.",
  "title": "Introducing DRM Language Emitter: Language Generation as Motion Through Learned Geometry"
}