{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiaienopwux43g523razebr7kekpugeyvyvnho3ygd6fb3en3ei32e",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3motmvbklvin2"
  },
  "path": "/t/shannon-prime-lattice/176466?page=2#post_36",
  "publishedAt": "2026-06-22T00:12:25.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "(click for more details)"
  ],
  "textContent": "## **The knowledge system (this repo owns the OKFS tooling)**\n\nSix months in, the binding constraint stopped being code and became _knowledge discipline_ — sessions kept rebuilding subsystems that already existed. The answer is a small, content-addressed knowledge layer, and this repo owns its tooling (`tools/okf_validate.py`, `tools/okf_mem.py`, `tools/okf_history.py`).\n\n  * **SP-OKF** (`papers/SP-OKF-PROFILE.md`) — Shannon-Prime’s profile of Google’s **Open Knowledge Format v0.1**. Every knowledge `.md` is a _concept_ with a controlled `type` + receipts-first frontmatter (`title/description/tags/timestamp/resource` + `sp_status/sp_gate/sp_commit/ sp_repro`). Cross-linked and validated by `tools/okf_validate.py` — gate **G-OKF-CONFORM** (currently 130 concepts, 0 errors, GREEN). New `type`s register in the profile §2 first.\n  * **MEM-OKF** (`papers/MEMORY-OKF-PROFILE.md`, `tools/okf_mem.py`, `memory-okf/`) — the content-addressed, tiered (**LUT → summary → full**) **anti-rebuild memory** , addressed by sha256 (text) / C2-LSH-sig (latent episode). One format for agent facts AND XBAR/NIGHTSHIFT episodes; the NIGHTSHIFT curator emits into it. **The`okf_mem lookup` pre-flight is binding: before building anything, look it up** — a new file for an existing capability is a defect. Verify with `python tools/okf_mem.py verify --root memory-okf` (gate `G-MEM-OKF-CONFORM`).\n  * **HISTORY** (`HISTORY.md`, generated by `tools/okf_history.py`) — a hashed MEM-OKF-style Tier-0 LUT of the last 80 commits: the git short-hash IS the content address, dig deeper via `git show <hash>` (git = the Tier-2 store).\n  * **AGENTS** (`AGENTS.md`) — the per-repo agent-navigation doc: read order, the pre-flight, the non-negotiables. Human + agent readable.\n\n\n\n\n    agent enters ─► AGENTS.md ─► prompt.md ─► PPT-LAT-STATE.md (proven)\n                                    │\n                                    ▼  PRE-FLIGHT (binding, before any build):\n                       okf_mem lookup --root memory-okf <kw>  +  grep the tree\n                                    │\n                ┌───────────────────┴────────────────────┐\n                ▼                                         ▼\n       memory-okf/LUT.md (Tier-0)              HISTORY.md (commit LUT)\n           │  follow addr                          │  git show <hash>\n           ▼                                       ▼\n       sum/<addr>.md → full/<addr>.md         full commit (Tier-2 = git)\n\n       every knowledge .md carries SP-OKF frontmatter → okf_validate.py (G-OKF-CONFORM)\n\n\n## **The system: a four-tier memory hierarchy plus a latent crossbar**\n\nThe original “two-ring” framing has grown into a four-tier hierarchy with an inter-model lane on top. Architecture ground truth lives in the lattice repo (`papers/RFC-XBAR-auditable-latent-crossbar.md`); this is the public map, each component tagged with its status.\n\nThe live recall path on the served 12B chat — _every write receipted, gated, and rewindable_ — at a glance:\n\n(click for more details)\n\ngated-GREEN is not GREEN-LIVE: a default-off flag is a null floor until set. The full box-by-box tier map is papers/STATUS-MAP-2026-06-21.md.\n\nLatest (2026-06-21) — the NIGHTSHIFT offline curator is gated-GREEN (synthetic) and the MEM-OKF anti-rebuild store is ACTIVE.\n\n  * run_kairos_curator (engine 6107f3e, default-off SP_NIGHTSHIFT_OFFLINE) closes the offline-consolidation loop on the 12B: a model-call ep.secret extractor → teacher-forced causal-ablation admit (TAU=−8) → conformant MEM-OKF emit. G-NIGHTSHIFT-CURATOR criteria 1-4 GREEN on the SYNTHETIC gate (novel “8-FALCON-7729” collapse −33.59 ACCEPT / parametric “Paris” 0.00 REJECT, ~33-nat separation; emit rc=0, addr-join verified). Criterion 5 (live B4 in-distribution on real chat turns) is PENDING — so this is gated-GREEN / default-off, not GREEN-LIVE like the served chat. The recall organism’s roles are fixed: the causal ablation oracle (TAU=−8) is the ADMISSION gate, the learned latent W_c head (SP_B3_WC) is the live RECALL selector, and the native Diffusion Judge stays in the drawer pending an OOD kill-test (it must beat W_c head-to-head first). MEM-OKF (tools/okf_mem.py + memory-okf/, spec papers/MEMORY-OKF-PROFILE.md) is the content-addressed LUT→summary→full store; its okf_mem lookup pre-flight is binding before building anything (see AGENTS.md). Record papers/CONTRACT-NIGHTSHIFT-CURATOR.md + papers/STATUS-MAP-2026-06-21.md.\n\n\n\nWhere this repo sits — the rings + XBAR\n\n  * XBAR (the auditable latent crossbar, lattice RFC-XBAR) is the system this substrate serves: an Exec (the big generator — the engine’s CUDA/CPU forwards) and a Memo (a small curator) share a tiered latent memory, and every write to canonical memory is receipted, gated, and rewindable. This repo owns the substrate tiers (Ring 1 + Ring 2 + the curator transaction\n\n  * the SP_REPLAY seam + the exact-integer container they sit on). The engine owns Exec’s accelerated forwards, the Optane/QUIC Ring-2 stores, the SP_XBAR_* harness, and the daemon.\n\n  * The recall-relevance problem the ARM contract posed — which stored episode is load-bearing for this query? — is SOLVED, but the live selector itself lives host-side in the engine daemon (recall.rs/routes.rs); NO frozen-ABI change and NO .sp-model format change — the L1 §6b verb and the OK_Q4 container are untouched. This core owns the episode store (core/arm/) + the exact-integer substrate it rides on; the selector is an engine-side rider. Detail in §5.\n\n\n\n\nThe recall organism — where the pieces live\nThe “which episode is load-bearing?” relevance problem the ARM contract posed is SOLVED, but the solution is split cleanly:\n\n  * This core owns the episode store (core/arm/ Ring-2 + the ARM router) and the exact-integer substrate it rides on.\n  * The engine owns the live selector. The fix is three host-side pieces (engine, default-off, null-floor): a curator (mints novel, non-parametric needles); a teacher-forced ablation labeler (SP_B3_SECRET cudaMemset-ablates the secret’s source KV rows and re-scores the secret’s NLL — novel needle collapse −33.56 vs parametric control −0.15, ~16-nat gap, pinned TAU=−8.0, the official ADMISSION oracle); and a learned W_c head trained on those labels (the live RECALL selector). Engine gate G-CHAT-B3-WC-DIV2 = 360/361 recall + 50/50 foreign-reject (int16==f32, s0=+0.102); LIVE G-CHAT-B3-WC-DEPLOY, engine edc8079.\n  * Architecture, fixed + recorded: the causal ablation oracle (TAU=−8) is the ADMISSION gate; the learned W_c head is the live RECALL selector; the native Diffusion Judge is UNPROVEN, in the drawer pending an OOD kill-test (its 95.6% is the external llama.cpp oracle’s number, not ours — the native single-forward was falsified ~25%; it must beat W_c head-to-head before earning deployment).\n\n\n\nThis rides entirely on this repo’s two-ring substrate but is an engine-side rider — the L1 §6b verb and the OK_Q4 .sp-model container are untouched. Detail: lattice CONTRACT-CHAT-FULLSTACK + SESSION-HANDOFF.md §0d.\n\nBoundary thesis — honest negatives\n\n  * O_K wins on EXACT ARITHMETIC (the container); every structure-on-content lever was measured-inert and is kept on the record as an [HONEST-NEGATIVE] (do not re-litigate):\n\n\n\nSplit-prime O_K Dirichlet carriers (d7d96fe) — operationally inert.\n\n  * Möbius-on-M (1e70763) — sheds memories 1.000 → 0.969 @ N=32.\n  * Entropy-coding the Frobenius codes (e6d17bb) — 1.02× dead weight (the lever is bit-width).\n  * T2-Möbius on the real 12B embedding (ac76c8e) — recon cos 0.032 == random.\n  * The compression reading of “byte-exact” — convicted redundant against the existing per-32-block OK_Q4B at gold PPL 4.6665. Byte-exact is the auditability axis only.\n  * NTT-attention is slower than fp32 dot at HD ≤ 256 (~0.15–0.72×); the substrate win is over HD (poly length), not ctx.\n  * KSTE is not a recall router (histogram = permutation-invariant; the directional ±1 Rademacher projection is the router). KSTE stays valid for dedup/dominance only.\n\n",
  "title": "Shannon Prime Lattice"
}