Context Gravity

After looking into it a bit, this is how I’d read it:…

Thanks for the detailed breakdown — this was genuinely useful. I implemented everything on the ablation list and have results.

What I built:

• SemanticForceProcessor as a proper LogitsProcessor using scores += log1p(force) — you’re right that the logit-space form is the natural HF interface
• Full ablation battery: real centroids, shuffled centroids, random centroids, no-IDF, uniform mass, local-only, universe+local, temperature, top-p, typical sampling
• Per-step diagnostic logging: force mean/max, active body count, top boosted tokens, escape rate

Results (GPT-2, 100 tokens, 8 prompts, matched temperature T=0.8):

What the ablation showed:

On geometry: You called it — shuffled centroids (same geometry, permuted mass assignments) match real almost exactly. But real vs random centroids does diverge: real perplexity 14.74 vs random 15.83, real distinct-1 0.493 vs random 0.505. The semantic universe doesn’t maximize diversity — it enforces topical coherence. Random centroids push toward arbitrary embedding regions, injecting broader but semantically unconstrained variety. The semantic field is a coherence enforcer, not a diversity maximizer.

On IDF: Removing IDF raises perplexity ~3.7% and increases distinct-1 ~1.8%. It contributes, but it’s not carrying the effect alone.

On the geometry being mostly flat: This one took the most debugging. With all 256 bodies active simultaneously, the force field across the vocabulary is nearly uniform — the force max/mean ratio was 3.1, not nearly selective enough for real vs random to diverge. The fix was top-k body selection: restrict force to the k=4 bodies most aligned with the current context position. With k=4, only semantically relevant bodies contribute, and force differentials become meaningful. Without this, real = shuffled = random.

On local bodies: Best perplexity of any gravitational condition (13.42), +11% distinct-1, runs at 43ms/token vs 93ms/token for the full universe. Pareto-optimal if latency matters.

One implementation gotcha worth flagging:

Temperature matching is a hard requirement for fair comparison. My initial benchmark passed raw logits to the gravitational sampler (effective T=1.0) while the baseline used T=0.8. This inflated perplexity by ~4x and made everything look broken. Once I matched temperature, the numbers above are what came out.

On the multiplicative vs logit-space distinction:

The logit-space form (log1p(force)) is what I implemented in SemanticForceProcessor — it’s the right HF interface. The distinction you noted is real though: multiplicative reweighting in probability space explicitly preserves near-zero probability tokens while logit-space application compresses that effect. In practice at the scale tested this didn’t produce visible differences, but it’s worth preserving in documentation for anyone who cares about the low-probability tail.

On “gravity replaces temperature”:

Fair point. The more accurate framing is: gravity adds a semantic-field reweighting term; top-k body selection makes that term geometrically selective; temperature still controls overall distribution sharpness. The two operate on orthogonal axes and combine naturally.

Code is in the repo. Happy to share the benchmark script directly if useful.