{
"$type": "site.standard.document",
"bskyPostRef": {
"cid": "bafyreigzbn6k3uwwvv5vzinohp2sybwubp2n23sm5jvp6z6n66vzjzoov4",
"uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mofktwngorr2"
},
"path": "/t/neon-city-cosysim-and-the-nexus-project/176853#post_1",
"publishedAt": "2026-06-16T08:36:52.000Z",
"site": "https://discuss.huggingface.co",
"tags": [
"Quickstart",
"docs/OPERATIONS.md",
"Overview & Architecture",
"docs/ARCHITECTURE.md",
"NEON CITY",
"Engine Internals",
"docs/MCP_FRAMEWORK.md",
"NLM + NEXUS",
"docs/NEXUS.md",
"CONTROL",
"Integrations, Apps & CLI",
"ARGUS",
"docs/ARGUS_METHODOLOGY.md",
"Creation Kit & Asset Studio",
"docs/DESIGN_SYSTEM_V2.md",
"docs/INDEX.md",
"LMStudio",
"ComfyUI",
"http://localhost:8500",
"(click for more details)",
"docs/SCENES.md",
"@skill",
"@mcp_tool"
],
"textContent": "OK, this is an old one. This started out as me trying to get realtime voice working 8 months ago. I ended up having to learn spec decoding, chunking, finetuned router models etc. I built up a 1100 test and telemety suite around it, Custom LMStudio clients and server etc. It eventually evolved into what you see today. There is ALOT more here than meets the eye. This is a complete self managed, self improving, self finetuning, self coding local management system with a vector DB with 6 cascades that leverages notebooklm in ways it isn’t exactly meant to be used, With an entire Scene Creation system/Simulator with drag and drop or even single prompt scene generation and asset studio. Every scene uses a local AI for something. All scenes are independant - yet act inside a living, breathing city with a real economy, day/night cycles, alliances, news cycles etc.\n\nEach agent has their own phone and can call, text, hack any other agent. You have The same phone. and much much more. It really is just a showcase of what is possible. All old credentials are expired/changed etc, so don’t bother. lol.\n\nAnyway, as usual If you any part useful or interesting, feel free to take and use or improve on yourself! This is presented here for one purpose only - Providing as much information and idea’s as possible to everyone so people can take inspiration or find something useful and advance the field in some way!\n\n# **CosySim**\n\n**A local-first, open AI simulation framework where every NPC is a real, governed LLM agent — and the world remembers.**\n\n35 launch targets · ~1,040 skills · 38-stage interceptor pipeline · 6-tier knowledge router · a training flywheel —\nbuilt almost entirely through **agentic coding** , and published so humans _and_ AI agents can learn from it.\n\n* * *\n\n> **Why this repo exists.** CosySim is meant to be _read_. It is a working, end-to-end example of what local agents + agentic coding can build: a living cyberpunk city whose residents reason on a local model, recall the past from a persistent knowledge base, react to a live economy and faction war, and quietly turn every interaction into training data that improves the next one. Take any piece you like — the interceptor pipeline, the LMStudio steering, the NLM↔Nexus flywheel, the ARGUS toolkit — and use it in your own project.\n\n## **Start here**\n\nPick the door that matches why you came:\n\n**You want to…** | **Go to** | **Deep-dive doc**\n---|---|---\n**Run it** in 5 minutes | Quickstart | docs/OPERATIONS.md\nUnderstand **how it fits together** | Overview & Architecture | docs/ARCHITECTURE.md\nSee the **game / living world** | NEON CITY | scene code in `content/scenes/`\nLearn **how agents are steered** | Engine Internals | docs/MCP_FRAMEWORK.md\nUnderstand the **AI brain** (local → frontier) | NLM + NEXUS | docs/NEXUS.md\n**Train / finetune / self-improve** | CONTROL | `engine/training/`, `training/`\nWire **external services** | Integrations, Apps & CLI | `docs/*_API_REFERENCE.md`\nDo **web-app reconnaissance** | ARGUS | docs/ARGUS_METHODOLOGY.md\n**Create** scenes / assets | Creation Kit & Asset Studio | docs/DESIGN_SYSTEM_V2.md\nBrowse **everything** | — | docs/INDEX.md\n\n## **Quickstart**\n\n> **Prerequisites:** Python 3.13, LMStudio running on `:1234` with a chat model loaded. Optional: ComfyUI (`:8188`) for image/video, a TTS server (`:8600`) for voice.\n\n\n # 1. Install\n pip install -r requirements.txt && npm install\n\n # 2. Configure secrets (nothing real is committed — see \"Security & configuration\")\n cp .env.example .env # then fill in any keys you have; LMStudio works with no auth\n\n # 3. Launch\n python tui.py # interactive Terminal UI (recommended) — ←/→/↑/↓ to navigate, Enter to launch\n python launcher.py --core # or: auto-start core services + main scenes\n python launcher.py neoncity # or: a single scene → http://localhost:5563\n python launcher.py --list # see all 35 targets + live port status\n\n\n\nThen open the hub at **http://localhost:8500** — the NEON CITY landing page — and jack in.\n\n\n # Handy\n python cli.py ask \"prompt\" # query the local model stack (38 models)\n python scripts/oracle.py # full system diagnostic (health · errors · perf)\n python scripts/smart_test.py --smoke # fast test sweep (~15 files)\n\n\n\n## **Security & configuration**\n\nThis repo is **safe to fork** : no live credentials are committed. Real secrets live only in gitignored local files.\n\n * **`.env`** (gitignored) — copy from **`.env.example`** and fill in what you have. Loaded automatically by `engine/config.py`. LMStudio needs no auth by default, so an empty `.env` is enough to run the world.\n * **`config/default.yaml`** — all tunables; secret-shaped values resolve from `${ENV_VAR}` at read time via a `SecretManager` hook.\n * **`config/nlm_rpcids.yaml`** (gitignored, runtime-regenerable) — Google rpc IDs + keys; ship-safe template is **`config/nlm_rpcids.example.yaml`** with every key redacted.\n * HARs, heap snapshots, dumps, cookies, account pools, and backups are all gitignored — capture intelligence stays local.\n\n\n\n* * *\n\n## **What is CosySim?**\n\n_The NEONCITY “Dark Renaissance” landing — the front door to a city of local agents._\n\n**CosySim is a local-first, open AI simulation framework where every NPC is a real, governed LLM agent — and the world they live in actually remembers.** It runs **35 launch targets** (18 game scenes + 11 services + 6 creation tools) as Flask/Socket.IO servers on your own machine, powered entirely by **local inference (LMStudio)** , a persistent knowledge layer (**Nexus KMS**), and **NotebookLM** distillation. No cloud API is required for core gameplay.\n\nIt is built to be read. This is a flagship example of what _agentic coding + local agents_ can produce: ~1,040 skills across 99 packs, a 36-stage interceptor pipeline, a 6-tier knowledge router, and a training flywheel — all wired together and documented so that **humans and AI agents alike can learn from it and borrow from it**.\n\n> The elevator pitch: _Spin up a neon cyberpunk city on your laptop. Talk to its residents. They reason with a local model, recall what you did last week from a persistent knowledge base, react to a live economy and faction war, and quietly turn every conversation into training data that makes the next conversation better._\n\n### **Why it’s unique**\n\n**Claim** | **How it’s actually true (in the code)**\n---|---\n**Every NPC is a real local agent** | Each reply runs through `AgentGovernor.reply()` in `engine/mcp/comms_framework.py` → `VirtualAgent` → `LMSClient.infer_stream()` against LMStudio on `localhost:1234`. No scripted dialog trees.\n**The city remembers** | State is a live tree (`MCPFramework` singleton: scenes → characters → world → factions). Persistent memory, rules, and 3.7K+ Q&A pairs live in **Nexus KMS** (SQLite + FTS5, `:8700`).\n**Frontier-level results from local models** | The 6-tier **NexusQueryRouter** answers from cache/vector/FTS first and only falls back to local inference last — then _auto-stores the answer_. **NotebookLM** (free Gemini, grounded) sits between as a distillation tier. Every interaction makes the next one cheaper and sharper.\n**It governs itself** | 36 interceptors (`engine/agents/interceptors/`) inject mood, knowledge, scene rules, faction standing, and heat awareness — then shape, log, and sync the response. Skills enforce cooldowns, costs, and prerequisites.\n**Open and inspectable** | Vanilla-JS frontend (no build step), Google-style docstrings, version-stamped change logs, and a deep `docs/` tree with `INDEX.md` as the door.\n\n## **Architecture overview**\n\nCosySim is a **reusable engine + swappable content + tunable config**. Scenes subclass `BaseScene`; the engine provides agents, governance, knowledge, world simulation, and inference; YAML config tunes behavior without touching code.\n\n\n ┌──────────────────────────────────────────────────────────────────────────┐\n │ BROWSER — Neon HUD v2 (vanilla JS · Jinja2 · Socket.IO client) │\n │ cosysim-telemetry.js · cosysim-particles.js · design_tokens.css (v2) │\n └─────────────────────────────────┬────────────────────────────────────────┘\n │ Socket.IO / REST\n ┌─────────────────────────────────▼────────────────────────────────────────┐\n │ 35 LAUNCH TARGETS (ports 5555–8800) │\n │ ┌── GAME (18) ──────┐ ┌── SERVICE (11) ─────┐ ┌── CREATION (6) ──────┐ │\n │ │ penthouse, neoncity│ │ nexus_kms, hub, │ │ canvas, asset_studio,│ │\n │ │ oracle, casino, │ │ tts, command_center,│ │ creation_kit, │ │\n │ │ heist, tavern, … │ │ intel_hub, … │ │ creator, … │ │\n │ └───────────────────┘ └─────────────────────┘ └──────────────────────┘ │\n └─────────────────────────────────┬────────────────────────────────────────┘\n │\n ┌─────────────────────────────────▼────────────────────────────────────────┐\n │ SKILLS + MCP PIPELINE │\n │ engine/skills/builtin/ ~1,040 @skill across 99 packs │\n │ engine/mcp/ MCPFramework state tree · 43 tool modules · DialogSystem │\n │ engine/agents/interceptors/ 36 pre/post hooks (AgentGovernor) │\n └─────────────────────────────────┬────────────────────────────────────────┘\n │\n ┌─────────────────────────────────▼────────────────────────────────────────┐\n │ ENGINE LAYER (engine/) │\n │ lmstudio/ ServerController · LMLink federation · StreamProcessor │\n │ nexus/ NexusClient · 6-tier QueryRouter · KnowledgePipeline · NLM │\n │ world/ WorldSim (economy ticks) · PlayerState · Missions · Crew │\n │ agents/ VirtualAgent · AgentGovernor · AgentRouter · OutputEvaluator │\n │ training/ DataCollector · BenchmarkRunner · FinetuneOrchestrator │\n └─────────────────────────────────┬────────────────────────────────────────┘\n │ external processes\n ┌─────────────────────────────────▼────────────────────────────────────────┐\n │ LMStudio :1234 · Nexus KMS :8700 · ComfyUI :8188 · TTS :8600 │\n │ (local LLM) (knowledge) (image/video) (Qwen3 voice) │\n └──────────────────────────────────────────────────────────────────────────┘\n\n\n\n\nThe engine subsystems map cleanly onto folders, so the diagram doubles as a directory map:\n\n**Layer** | **Where** | **Does what**\n---|---|---\n**Browser HUD** | `content/shared/`, scene `templates/` + `static/` | Neon HUD v2, particles, telemetry — pure vanilla JS, no build step\n**Targets** | `content/scenes/{name}/`, `engine/control_plane_registry.py` | 35 Flask/FastAPI/Streamlit/Node servers, each a `BaseScene` subclass\n**Skills + MCP** | `engine/skills/`, `engine/mcp/` | `@skill` capabilities, the `MCPFramework` state tree, 43 `@mcp_tool` modules, governance\n**Engine** | `engine/lmstudio` · `nexus` · `world` · `agents` · `training` | inference, knowledge, simulation, agent lifecycle, the data flywheel\n**External** | LMStudio · Nexus KMS · ComfyUI · TTS | Local inference, knowledge backbone, media generation, voice\n\n## **How a player message becomes an NPC reply**\n\nThis is the heart of the system — the path a single message takes is the same for every NPC in every scene. It is implemented in `AgentGovernor.reply()` (`engine/mcp/comms_framework.py`) and the interceptor pipeline.\n\n\n Player message ──Socket.IO / REST──► Scene ──► DialogSystem.add_turn()\n │\n ▼\n ┌─ AgentGovernor.reply() ────────────────────────────────────────────────┐\n │ │\n │ 1. Build ResponseContext (scene, agent_id, user_message, manifest) │\n │ │\n │ 2. AUTO skills run BEFORE the LLM; cooldown + prereq gated │\n │ e.g. search_memory → result injected into context │\n │ │\n │ 3. run_pre() ─ ~21 PRE interceptors, ascending priority ───────────► │\n │ 5 MoodDrift · 6 NexusPrompt (hydrate knowledge) │\n │ 8 CharRegistry · 10 Router (pick model) · 15 Scene context │\n │ 40 FactionContext · 50 PersonalityGuard · 60 PolicyEnforcer │\n │ (any PRE interceptor may set ctx[\"abort\"] / ctx[\"skip_llm\"]) │\n │ │\n │ 4. LLM CALL VirtualAgent.reply(governance_context) │\n │ └► LMSClient.infer_stream() ──SSE──► LMStudio :1234 │\n │ │\n │ 5. Parse StreamProcessor / ContentRouter extract inline tags: │\n │ [MOOD:happy] [IMAGE:prompt] [ACTION:x] [STAT:health+10] [VOICE] │\n │ │\n │ 6. run_post() ─ ~7 POST interceptors ─────────────────────────────► │\n │ 75 HeatAwareness · 80 ResponseShaper · 85 TTS · 90 Logger │\n │ 92 MoodSync/SpectatorBroadcast · 93 Relationship │\n └─────────────────────────────────────┬───────────────────────────────────┘\n ▼\n STATE WRITES + CASCADE\n • CharacterRegistry mood / relationship deltas\n • PlayerState credits / rep / heat ([STAT:…] tags)\n • ComfyUI image queue ([IMAGE:…] tags)\n • OutputEvaluator.score → DataCollector (training flywheel)\n • WorldSim / EventCascade → broadcast to other subscribed scenes\n ▼\n Scene emits via Socket.IO: chat message · HUD update · portrait mood · TTS audio\n\n\n\n\nA few details that make this more than a wrapper:\n\n * **`ResponseContext` is a mutable bag** passed to every interceptor — `system_prompt`, `messages`, `reply`, `auto_results`, plus post-call metadata like `mood_tags`, `response_id`, and `is_stateful`. Interceptors read and write it in priority order; any PRE interceptor can abort the LLM call entirely.\n * **Skills have three trigger types** (`engine/mcp/comms_framework.py`): `auto` (run before the call, result injected), `optional` (offered to the model as a tool), and `required` (the model must call it). Auto-skill invocation is throttled by a real `COOLDOWN_TRACKER` and prerequisite chain (v1.59.0).\n * **Stream tags are the action channel.** The LLM doesn’t just talk — it emits `[STAT:health+10]`, `[MOOD:x]`, `[IMAGE:prompt]` inline, parsed by `StreamProcessor` (`_RE_STAT = \\[STAT:(\\w+)([+-]\\d+)\\]`) and routed to game state, mood, and the ComfyUI queue.\n * **The cascade is what makes the city feel alive.** A `WorldSim` tick (~60s) emits `SimEvent`s (economy / faction shift / NPC action / weather); `EventCascade` broadcasts to every subscribed scene, so an NPC’s action in one room ripples through the HUD and faction standings everywhere.\n\nThe self-improving loop (why local models punch above their weight) (click for more details)\n\n* * *\n\n## **NEON CITY — the living world & game mechanics**\n\n> _The city breathes. Six factions fight for control. The night never ends._ — `content/scenes/neoncity/neoncity_scene.py`\n\nNEON CITY is CosySim’s flagship: not a single screen but a **persistent, autonomous world** that the GAME-pillar scenes all plug into. The economy keeps moving, factions keep fighting over turf, NPCs keep walking their daily routines, and the weather keeps rolling through neon-soaked rain — whether or not a player is logged in. When you _do_ walk in, the world has a memory: your last heist raised the heat, your faction standing changes the prices you’re quoted, and a botched job last night is still rippling through three other scenes.\n\nThe whole thing runs on **local inference** (LMStudio). No cloud, no API keys for the simulation itself — a city full of agents reasoning on your own GPU.\n\n### **One world, many windows**\n\nThe GAME pillar is a set of independent Flask/Socket.IO scenes (each on its own port, inheriting `BaseScene`), but they are **windows into the same world state** , not separate games. NEON CITY (`:5563`) is the hub; the rest are districts and venues you move through.\n\n**Scene** | **Display name** | **What it is**\n---|---|---\n`neoncity` | **NEON CITY** | Living-world hub: economy, factions, missions, crew, cyberspace, board mode\n`phone` | **SIGNAL** | Cyberdeck — messaging, 0xGH0ST contacts, mini-games, the always-on HUD\n`penthouse` | **THE PENTHOUSE** | 3D character room (three.js r184), curtain-wall skyline, autonomous NPCs\n`heist` | **THE HEIST** | Crew-driven heist runs that feed heat and faction consequences back into the city\n`arena` | **THE COLOSSEUM** | Combat matches; bookmaking and fighter queues spawned by the world sim\n`casino` | **CLUB NOIR** | Gambling, VIP access gated by faction standing\n`lounge` | **THE VELVET PIT** | Speakeasy social scene, ambient events, brokered deals\n`tavern` | **THE RUSTY ANCHOR** | Fantasy RPG tavern, barter economy, dice\n`realm` | LitRPG world | Dual-agent (Director + companion), d20 combat, inventory\n`grid` | **THE GRID** | District board / strategy layer over territory control\n`games` | **THE ARCADE** | Trivia, chess puzzles, leaderboards, AI opponents\n… | | plus `gallery`, `cyberspace`, and the broader GAME catalogue\n\nThe authoritative catalogue lives in `engine/control_plane_registry.py` (resolved to ports by `engine/port_registry.py`); the launcher, TUI, and the in-browser **THE TERMINAL** catalogue all derive their lists from it. See docs/SCENES.md for the full pillar tables.\n\n### **The living world: two coordinated daemons**\n\nNEON CITY’s “aliveness” comes from background daemon threads in `engine/world/` that tick on a game clock (**1 real second ≈ 1 game minute; 60 real seconds = 1 game hour**).\n\n**`WorldSim`** (`engine/world/world_sim.py`) is the event engine. It fires scripted-but-stochastic events on per-task intervals — NPC actions every ~60s, ambient mood shifts, faction shifts, 0xGH0ST hacker messages, arena match queues, major world events, and passive economy ticks. Every `SimEvent` is (a) appended to a 200-entry ring buffer, (b) persisted to **Nexus KMS** as `world_sim` history, and (c) broadcast on the `EventBus`. A `get_digest(scene)` call returns “what you missed” the moment you enter a scene.\n\n**`LivingWorld`** (`engine/world/living_world.py`) is the orchestrator. Each tick it coordinates every subsystem in order:\n\n\n 1. Game clock → read time-of-day from WorldState\n 2. NPC routines → RoutineManager moves NPCs to scheduled locations\n 3. Faction AI → each faction makes one strategic decision (every 5th tick)\n 4. Market tick → supply/demand drift, territory-weighted prices\n 5. Weather cycle → Markov transition (CLEAR→NEON_RAIN→STORM→BLACKOUT…)\n 6. Stochastic events → fire + propagate consequences to market & NPCs\n\n\n\n\nThe subsystems, by file (click for more details)\n\n### **“The city remembers” — the v1.59 / v1.60 feedback loops**\n\nEarlier versions _simulated_ a world but the simulation was largely cosmetic — agents could say `[STAT:arousal+10]`, a faction could “expand territory”, and **nothing actually changed**. The v1.59 _“consequential world”_ and v1.60 _“Living Systems”_ passes closed those loops. This is the part worth studying: it’s where a pile of independent systems became a world with cause and effect.\n\n**Loop** | **Before** | **After (the closed loop)** | **Where**\n---|---|---|---\n**Stat tags applied** | `[STAT:trust-5]` parsed then discarded | `StatSyncInterceptor` (priority 91) writes tags to real character state _before_ mood rules evaluate them | `engine/agents/interceptors/stat_sync.py`\n**Economy settlement** | Buying/selling was flavour text | `Market._settle_buy/_settle_sell` debit the wallet, move inventory, and raise heat for illegal goods | `engine/world/market.py`\n**World→market shocks** | World events never reached the market | `Market.subscribe_to_world_events()` maps `gang_war→weapons up`, `festival→luxury up`, `shortage→category surge` | `living_world._init_subsystems` → `market.py`\n**Faction gating** | Standing only gated casino VIP | `faction_gates` gives every scene shop access, ally discounts (−10%), rival surcharges (+15%), and standing-locked missions | `engine/world/faction_gates.py`\n**Player-aware factions** | Factions ignored the player | Faction AI weights decisions by _your_ standing — allies expand near you, rivals raid your turf; shifts broadcast on `NEONCITY_FACTION_SHIFT` | `engine/world/faction_ai.py`\n**Mission consequences** | Missions resolved in isolation | `MissionManager._apply_consequences` applies rep/heat/faction-control deltas on success _and_ failure; chains branch on outcome & standing | `mission.py` + `mission_chains.py`\n**Equipment matters** | Equipped gear gave +0 | `get_equipment_bonuses` aggregates equipped items into skill/stat deltas that skill-checks consume | `engine/world/equipment_effects.py`\n**Cross-scene ripples** | Events stayed local | `EventCascade` fans `WorldSim` events to subscribed scenes (e.g. a `CRIME` event reaches `phone`, `tavern`, `casino`, `heist`) | `engine/world/event_cascade.py`\n\nThe net effect: a heist that goes wrong raises city-wide heat, which `HeatAwarenessInterceptor` (pipeline priority 75) makes NPCs _aware_ of; the resulting faction shift reprices the black market; and the next mission in the chain unlocks a different branch because your standing changed. Every magic number here is config-driven (`mission.consequences.*`, `economy.event_shocks.*`, `territory.faction_ai.*`) so the whole consequence economy is tunable without touching code.",
"title": "NEON-CITY/CosySim and the NEXUS project"
}