{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiegod4dcc3hqpsedxror3xspxwtgniqujfhv42f43gk4ksspzgs7u",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mptrlverxut2"
  },
  "path": "/t/context-gravity/177329#post_13",
  "publishedAt": "2026-07-04T18:12:31.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "> After looking into it a bit, this is how I’d read it:…\n\nThanks for the detailed breakdown — this was genuinely useful. I implemented everything on the ablation list and have results.\n\n**What I built:**\n\n  * `SemanticForceProcessor` as a proper `LogitsProcessor` using `scores += log1p(force)` — you’re right that the logit-space form is the natural HF interface\n  * Full ablation battery: real centroids, shuffled centroids, random centroids, no-IDF, uniform mass, local-only, universe+local, temperature, top-p, typical sampling\n  * Per-step diagnostic logging: force mean/max, active body count, top boosted tokens, escape rate\n\n\n\n**Results (GPT-2, 100 tokens, 8 prompts, matched temperature T=0.8):**\n\nCondition | Perplexity | Distinct-1 | Rep-2\n---|---|---|---\nTemperature T=0.8 | 9.84 | 0.411 | 0.187\nTypical sampling | 11.64 | 0.427 | 0.141\nLocal bodies only | 13.42 | 0.457 | 0.174\nUniverse (real) | 14.74 | 0.493 | 0.108\nUniverse (shuffled) | 14.76 | 0.494 | —\nUniverse (random) | 15.83 | 0.505 | 0.116\nTop-p (p=0.95) | 6.67 | 0.350 | —\n\n**What the ablation showed:**\n\n_On geometry:_ You called it — shuffled centroids (same geometry, permuted mass assignments) match real almost exactly. But real vs random centroids does diverge: real perplexity 14.74 vs random 15.83, real distinct-1 0.493 vs random 0.505. The semantic universe doesn’t maximize diversity — it enforces topical coherence. Random centroids push toward arbitrary embedding regions, injecting broader but semantically unconstrained variety. The semantic field is a coherence enforcer, not a diversity maximizer.\n\n_On IDF:_ Removing IDF raises perplexity ~3.7% and increases distinct-1 ~1.8%. It contributes, but it’s not carrying the effect alone.\n\n_On the geometry being mostly flat:_ This one took the most debugging. With all 256 bodies active simultaneously, the force field across the vocabulary is nearly uniform — the force max/mean ratio was 3.1, not nearly selective enough for real vs random to diverge. The fix was top-k body selection: restrict force to the k=4 bodies most aligned with the current context position. With k=4, only semantically relevant bodies contribute, and force differentials become meaningful. Without this, real = shuffled = random.\n\n_On local bodies:_ Best perplexity of any gravitational condition (13.42), +11% distinct-1, runs at 43ms/token vs 93ms/token for the full universe. Pareto-optimal if latency matters.\n\n**One implementation gotcha worth flagging:**\n\nTemperature matching is a hard requirement for fair comparison. My initial benchmark passed raw logits to the gravitational sampler (effective T=1.0) while the baseline used T=0.8. This inflated perplexity by ~4x and made everything look broken. Once I matched temperature, the numbers above are what came out.\n\n**On the multiplicative vs logit-space distinction:**\n\nThe logit-space form (`log1p(force)`) is what I implemented in `SemanticForceProcessor` — it’s the right HF interface. The distinction you noted is real though: multiplicative reweighting in probability space explicitly preserves near-zero probability tokens while logit-space application compresses that effect. In practice at the scale tested this didn’t produce visible differences, but it’s worth preserving in documentation for anyone who cares about the low-probability tail.\n\n**On “gravity replaces temperature”:**\n\nFair point. The more accurate framing is: gravity adds a semantic-field reweighting term; top-k body selection makes that term geometrically selective; temperature still controls overall distribution sharpness. The two operate on orthogonal axes and combine naturally.\n\nCode is in the repo. Happy to share the benchmark script directly if useful.",
  "title": "Context Gravity"
}