Shannon Prime Lattice

The knowledge system (this repo owns the OKFS tooling)

Six months in, the binding constraint stopped being code and became knowledge discipline — sessions kept rebuilding subsystems that already existed. The answer is a small, content-addressed knowledge layer, and this repo owns its tooling (tools/okf_validate.py, tools/okf_mem.py, tools/okf_history.py).

• SP-OKF (papers/SP-OKF-PROFILE.md) — Shannon-Prime’s profile of Google’s Open Knowledge Format v0.1. Every knowledge .md is a concept with a controlled type + receipts-first frontmatter (title/description/tags/timestamp/resource + sp_status/sp_gate/sp_commit/ sp_repro). Cross-linked and validated by tools/okf_validate.py — gate G-OKF-CONFORM (currently 130 concepts, 0 errors, GREEN). New types register in the profile §2 first.

• MEM-OKF (papers/MEMORY-OKF-PROFILE.md, tools/okf_mem.py, memory-okf/) — the content-addressed, tiered (LUT → summary → full) anti-rebuild memory , addressed by sha256 (text) / C2-LSH-sig (latent episode). One format for agent facts AND XBAR/NIGHTSHIFT episodes; the NIGHTSHIFT curator emits into it. Theokf_mem lookup pre-flight is binding: before building anything, look it up — a new file for an existing capability is a defect. Verify with python tools/okf_mem.py verify --root memory-okf (gate G-MEM-OKF-CONFORM).

• HISTORY (HISTORY.md, generated by tools/okf_history.py) — a hashed MEM-OKF-style Tier-0 LUT of the last 80 commits: the git short-hash IS the content address, dig deeper via git show <hash> (git = the Tier-2 store).

• AGENTS (AGENTS.md) — the per-repo agent-navigation doc: read order, the pre-flight, the non-negotiables. Human + agent readable.

  agent enters ─► AGENTS.md ─► prompt.md ─► PPT-LAT-STATE.md (proven) │ ▼ PRE-FLIGHT (binding, before any build): okf_mem lookup --root memory-okf + grep the tree │ ┌───────────────────┴────────────────────┐ ▼ ▼ memory-okf/LUT.md (Tier-0) HISTORY.md (commit LUT) │ follow addr │ git show ▼ ▼ sum/.md → full/.md full commit (Tier-2 = git)

  every knowledge .md carries SP-OKF frontmatter → okf_validate.py (G-OKF-CONFORM)

The system: a four-tier memory hierarchy plus a latent crossbar

The original “two-ring” framing has grown into a four-tier hierarchy with an inter-model lane on top. Architecture ground truth lives in the lattice repo (papers/RFC-XBAR-auditable-latent-crossbar.md); this is the public map, each component tagged with its status.

The live recall path on the served 12B chat — every write receipted, gated, and rewindable — at a glance:

(click for more details)

gated-GREEN is not GREEN-LIVE: a default-off flag is a null floor until set. The full box-by-box tier map is papers/STATUS-MAP-2026-06-21.md.

Latest (2026-06-21) — the NIGHTSHIFT offline curator is gated-GREEN (synthetic) and the MEM-OKF anti-rebuild store is ACTIVE.

• run_kairos_curator (engine 6107f3e, default-off SP_NIGHTSHIFT_OFFLINE) closes the offline-consolidation loop on the 12B: a model-call ep.secret extractor → teacher-forced causal-ablation admit (TAU=−8) → conformant MEM-OKF emit. G-NIGHTSHIFT-CURATOR criteria 1-4 GREEN on the SYNTHETIC gate (novel “8-FALCON-7729” collapse −33.59 ACCEPT / parametric “Paris” 0.00 REJECT, ~33-nat separation; emit rc=0, addr-join verified). Criterion 5 (live B4 in-distribution on real chat turns) is PENDING — so this is gated-GREEN / default-off, not GREEN-LIVE like the served chat. The recall organism’s roles are fixed: the causal ablation oracle (TAU=−8) is the ADMISSION gate, the learned latent W_c head (SP_B3_WC) is the live RECALL selector, and the native Diffusion Judge stays in the drawer pending an OOD kill-test (it must beat W_c head-to-head first). MEM-OKF (tools/okf_mem.py + memory-okf/, spec papers/MEMORY-OKF-PROFILE.md) is the content-addressed LUT→summary→full store; its okf_mem lookup pre-flight is binding before building anything (see AGENTS.md). Record papers/CONTRACT-NIGHTSHIFT-CURATOR.md + papers/STATUS-MAP-2026-06-21.md.

Where this repo sits — the rings + XBAR

• XBAR (the auditable latent crossbar, lattice RFC-XBAR) is the system this substrate serves: an Exec (the big generator — the engine’s CUDA/CPU forwards) and a Memo (a small curator) share a tiered latent memory, and every write to canonical memory is receipted, gated, and rewindable. This repo owns the substrate tiers (Ring 1 + Ring 2 + the curator transaction

• the SP_REPLAY seam + the exact-integer container they sit on). The engine owns Exec’s accelerated forwards, the Optane/QUIC Ring-2 stores, the SP_XBAR_* harness, and the daemon.

• The recall-relevance problem the ARM contract posed — which stored episode is load-bearing for this query? — is SOLVED, but the live selector itself lives host-side in the engine daemon (recall.rs/routes.rs); NO frozen-ABI change and NO .sp-model format change — the L1 §6b verb and the OK_Q4 container are untouched. This core owns the episode store (core/arm/) + the exact-integer substrate it rides on; the selector is an engine-side rider. Detail in §5.

The recall organism — where the pieces live The “which episode is load-bearing?” relevance problem the ARM contract posed is SOLVED, but the solution is split cleanly:

• This core owns the episode store (core/arm/ Ring-2 + the ARM router) and the exact-integer substrate it rides on.
• The engine owns the live selector. The fix is three host-side pieces (engine, default-off, null-floor): a curator (mints novel, non-parametric needles); a teacher-forced ablation labeler (SP_B3_SECRET cudaMemset-ablates the secret’s source KV rows and re-scores the secret’s NLL — novel needle collapse −33.56 vs parametric control −0.15, ~16-nat gap, pinned TAU=−8.0, the official ADMISSION oracle); and a learned W_c head trained on those labels (the live RECALL selector). Engine gate G-CHAT-B3-WC-DIV2 = 360/361 recall + 50/50 foreign-reject (int16==f32, s0=+0.102); LIVE G-CHAT-B3-WC-DEPLOY, engine edc8079.
• Architecture, fixed + recorded: the causal ablation oracle (TAU=−8) is the ADMISSION gate; the learned W_c head is the live RECALL selector; the native Diffusion Judge is UNPROVEN, in the drawer pending an OOD kill-test (its 95.6% is the external llama.cpp oracle’s number, not ours — the native single-forward was falsified ~25%; it must beat W_c head-to-head before earning deployment).

This rides entirely on this repo’s two-ring substrate but is an engine-side rider — the L1 §6b verb and the OK_Q4 .sp-model container are untouched. Detail: lattice CONTRACT-CHAT-FULLSTACK + SESSION-HANDOFF.md §0d.

Boundary thesis — honest negatives

• O_K wins on EXACT ARITHMETIC (the container); every structure-on-content lever was measured-inert and is kept on the record as an [HONEST-NEGATIVE] (do not re-litigate):

Split-prime O_K Dirichlet carriers (d7d96fe) — operationally inert.

• Möbius-on-M (1e70763) — sheds memories 1.000 → 0.969 @ N=32.
• Entropy-coding the Frobenius codes (e6d17bb) — 1.02× dead weight (the lever is bit-width).
• T2-Möbius on the real 12B embedding (ac76c8e) — recon cos 0.032 == random.
• The compression reading of “byte-exact” — convicted redundant against the existing per-32-block OK_Q4B at gold PPL 4.6665. Byte-exact is the auditability axis only.
• NTT-attention is slower than fp32 dot at HD ≤ 256 (~0.15–0.72×); the substrate win is over HD (poly length), not ctx.
• KSTE is not a recall router (histogram = permutation-invariant; the directional ±1 Rademacher projection is the router). KSTE stays valid for dedup/dominance only.