External Publication

NEON-CITY/CosySim and the NEXUS project

Hugging Face Forums [Unofficial] June 16, 2026

OK, this is an old one. This started out as me trying to get realtime voice working 8 months ago. I ended up having to learn spec decoding, chunking, finetuned router models etc. I built up a 1100 test and telemety suite around it, Custom LMStudio clients and server etc. It eventually evolved into what you see today. There is ALOT more here than meets the eye. This is a complete self managed, self improving, self finetuning, self coding local management system with a vector DB with 6 cascades that leverages notebooklm in ways it isn’t exactly meant to be used, With an entire Scene Creation system/Simulator with drag and drop or even single prompt scene generation and asset studio. Every scene uses a local AI for something. All scenes are independant - yet act inside a living, breathing city with a real economy, day/night cycles, alliances, news cycles etc.

Each agent has their own phone and can call, text, hack any other agent. You have The same phone. and much much more. It really is just a showcase of what is possible. All old credentials are expired/changed etc, so don’t bother. lol.

Anyway, as usual If you any part useful or interesting, feel free to take and use or improve on yourself! This is presented here for one purpose only - Providing as much information and idea’s as possible to everyone so people can take inspiration or find something useful and advance the field in some way!

CosySim

A local-first, open AI simulation framework where every NPC is a real, governed LLM agent — and the world remembers.

35 launch targets · ~1,040 skills · 38-stage interceptor pipeline · 6-tier knowledge router · a training flywheel — built almost entirely through agentic coding , and published so humans and AI agents can learn from it.

Why this repo exists. CosySim is meant to be read. It is a working, end-to-end example of what local agents + agentic coding can build: a living cyberpunk city whose residents reason on a local model, recall the past from a persistent knowledge base, react to a live economy and faction war, and quietly turn every interaction into training data that improves the next one. Take any piece you like — the interceptor pipeline, the LMStudio steering, the NLM↔Nexus flywheel, the ARGUS toolkit — and use it in your own project.

Start here

Pick the door that matches why you came:

You want to…	Go to	Deep-dive doc
Run it in 5 minutes	Quickstart	docs/OPERATIONS.md
Understand how it fits together	Overview & Architecture	docs/ARCHITECTURE.md
See the game / living world	NEON CITY	scene code in `content/scenes/`
Learn how agents are steered	Engine Internals	docs/MCP_FRAMEWORK.md
Understand the AI brain (local → frontier)	NLM + NEXUS	docs/NEXUS.md
Train / finetune / self-improve	CONTROL	`engine/training/`, `training/`
Wire external services	Integrations, Apps & CLI	`docs/*_API_REFERENCE.md`
Do web-app reconnaissance	ARGUS	docs/ARGUS_METHODOLOGY.md
Create scenes / assets	Creation Kit & Asset Studio	docs/DESIGN_SYSTEM_V2.md
Browse everything	—	docs/INDEX.md

Quickstart

Prerequisites: Python 3.13, LMStudio running on :1234 with a chat model loaded. Optional: ComfyUI (:8188) for image/video, a TTS server (:8600) for voice.

# 1. Install
pip install -r requirements.txt && npm install

# 2. Configure secrets (nothing real is committed — see "Security & configuration")
cp .env.example .env          # then fill in any keys you have; LMStudio works with no auth

# 3. Launch
python tui.py                 # interactive Terminal UI (recommended) — ←/→/↑/↓ to navigate, Enter to launch
python launcher.py --core     # or: auto-start core services + main scenes
python launcher.py neoncity   # or: a single scene → http://localhost:5563
python launcher.py --list     # see all 35 targets + live port status

Then open the hub at http://localhost:8500 — the NEON CITY landing page — and jack in.

# Handy
python cli.py ask "prompt"           # query the local model stack (38 models)
python scripts/oracle.py             # full system diagnostic (health · errors · perf)
python scripts/smart_test.py --smoke # fast test sweep (~15 files)

Security & configuration

This repo is safe to fork : no live credentials are committed. Real secrets live only in gitignored local files.

.env (gitignored) — copy from .env.example and fill in what you have. Loaded automatically by engine/config.py. LMStudio needs no auth by default, so an empty .env is enough to run the world.
config/default.yaml — all tunables; secret-shaped values resolve from ${ENV_VAR} at read time via a SecretManager hook.
config/nlm_rpcids.yaml (gitignored, runtime-regenerable) — Google rpc IDs + keys; ship-safe template is config/nlm_rpcids.example.yaml with every key redacted.
HARs, heap snapshots, dumps, cookies, account pools, and backups are all gitignored — capture intelligence stays local.

What is CosySim?

The NEONCITY “Dark Renaissance” landing — the front door to a city of local agents.

CosySim is a local-first, open AI simulation framework where every NPC is a real, governed LLM agent — and the world they live in actually remembers. It runs 35 launch targets (18 game scenes + 11 services + 6 creation tools) as Flask/Socket.IO servers on your own machine, powered entirely by local inference (LMStudio) , a persistent knowledge layer (Nexus KMS), and NotebookLM distillation. No cloud API is required for core gameplay.

It is built to be read. This is a flagship example of what agentic coding + local agents can produce: ~1,040 skills across 99 packs, a 36-stage interceptor pipeline, a 6-tier knowledge router, and a training flywheel — all wired together and documented so that humans and AI agents alike can learn from it and borrow from it.

The elevator pitch: Spin up a neon cyberpunk city on your laptop. Talk to its residents. They reason with a local model, recall what you did last week from a persistent knowledge base, react to a live economy and faction war, and quietly turn every conversation into training data that makes the next conversation better.

Why it’s unique

Claim	How it’s actually true (in the code)
Every NPC is a real local agent	Each reply runs through `AgentGovernor.reply()` in `engine/mcp/comms_framework.py` → `VirtualAgent` → `LMSClient.infer_stream()` against LMStudio on `localhost:1234`. No scripted dialog trees.
The city remembers	State is a live tree (`MCPFramework` singleton: scenes → characters → world → factions). Persistent memory, rules, and 3.7K+ Q&A pairs live in Nexus KMS (SQLite + FTS5, `:8700`).
Frontier-level results from local models	The 6-tier NexusQueryRouter answers from cache/vector/FTS first and only falls back to local inference last — then auto-stores the answer. NotebookLM (free Gemini, grounded) sits between as a distillation tier. Every interaction makes the next one cheaper and sharper.
It governs itself	36 interceptors (`engine/agents/interceptors/`) inject mood, knowledge, scene rules, faction standing, and heat awareness — then shape, log, and sync the response. Skills enforce cooldowns, costs, and prerequisites.
Open and inspectable	Vanilla-JS frontend (no build step), Google-style docstrings, version-stamped change logs, and a deep `docs/` tree with `INDEX.md` as the door.

Architecture overview

CosySim is a reusable engine + swappable content + tunable config. Scenes subclass BaseScene; the engine provides agents, governance, knowledge, world simulation, and inference; YAML config tunes behavior without touching code.

┌──────────────────────────────────────────────────────────────────────────┐
│  BROWSER — Neon HUD v2  (vanilla JS · Jinja2 · Socket.IO client)          │
│  cosysim-telemetry.js · cosysim-particles.js · design_tokens.css (v2)     │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │  Socket.IO / REST
┌─────────────────────────────────▼────────────────────────────────────────┐
│  35 LAUNCH TARGETS  (ports 5555–8800)                                     │
│  ┌── GAME (18) ──────┐ ┌── SERVICE (11) ─────┐ ┌── CREATION (6) ──────┐   │
│  │ penthouse, neoncity│ │ nexus_kms, hub,     │ │ canvas, asset_studio,│   │
│  │ oracle, casino,    │ │ tts, command_center,│ │ creation_kit,        │   │
│  │ heist, tavern, …   │ │ intel_hub, …        │ │ creator, …           │   │
│  └───────────────────┘ └─────────────────────┘ └──────────────────────┘   │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │
┌─────────────────────────────────▼────────────────────────────────────────┐
│  SKILLS  +  MCP PIPELINE                                                   │
│  engine/skills/builtin/  ~1,040 @skill across 99 packs                     │
│  engine/mcp/  MCPFramework state tree · 43 tool modules · DialogSystem     │
│  engine/agents/interceptors/  36 pre/post hooks (AgentGovernor)           │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │
┌─────────────────────────────────▼────────────────────────────────────────┐
│  ENGINE LAYER  (engine/)                                                   │
│   lmstudio/  ServerController · LMLink federation · StreamProcessor        │
│   nexus/     NexusClient · 6-tier QueryRouter · KnowledgePipeline · NLM    │
│   world/     WorldSim (economy ticks) · PlayerState · Missions · Crew      │
│   agents/    VirtualAgent · AgentGovernor · AgentRouter · OutputEvaluator  │
│   training/  DataCollector · BenchmarkRunner · FinetuneOrchestrator        │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │  external processes
┌─────────────────────────────────▼────────────────────────────────────────┐
│  LMStudio :1234  ·  Nexus KMS :8700  ·  ComfyUI :8188  ·  TTS :8600       │
│  (local LLM)        (knowledge)         (image/video)     (Qwen3 voice)    │
└──────────────────────────────────────────────────────────────────────────┘

The engine subsystems map cleanly onto folders, so the diagram doubles as a directory map:

Layer	Where	Does what
Browser HUD	`content/shared/`, scene `templates/` + `static/`	Neon HUD v2, particles, telemetry — pure vanilla JS, no build step
Targets	`content/scenes/{name}/`, `engine/control_plane_registry.py`	35 Flask/FastAPI/Streamlit/Node servers, each a `BaseScene` subclass
Skills + MCP	`engine/skills/`, `engine/mcp/`	`@skill` capabilities, the `MCPFramework` state tree, 43 `@mcp_tool` modules, governance
Engine	`engine/lmstudio` · `nexus` · `world` · `agents` · `training`	inference, knowledge, simulation, agent lifecycle, the data flywheel
External	LMStudio · Nexus KMS · ComfyUI · TTS	Local inference, knowledge backbone, media generation, voice

How a player message becomes an NPC reply

This is the heart of the system — the path a single message takes is the same for every NPC in every scene. It is implemented in AgentGovernor.reply() (engine/mcp/comms_framework.py) and the interceptor pipeline.

Player message  ──Socket.IO / REST──►  Scene  ──►  DialogSystem.add_turn()
        │
        ▼
 ┌─ AgentGovernor.reply() ────────────────────────────────────────────────┐
 │                                                                         │
 │  1. Build ResponseContext   (scene, agent_id, user_message, manifest)   │
 │                                                                         │
 │  2. AUTO skills             run BEFORE the LLM; cooldown + prereq gated  │
 │       e.g. search_memory → result injected into context                 │
 │                                                                         │
 │  3. run_pre()  ─ ~21 PRE interceptors, ascending priority ───────────►  │
 │       5  MoodDrift  ·  6 NexusPrompt (hydrate knowledge)                 │
 │       8  CharRegistry  ·  10 Router (pick model)  ·  15 Scene context    │
 │       40 FactionContext  ·  50 PersonalityGuard  ·  60 PolicyEnforcer    │
 │       (any PRE interceptor may set ctx["abort"] / ctx["skip_llm"])       │
 │                                                                         │
 │  4. LLM CALL   VirtualAgent.reply(governance_context)                    │
 │                  └► LMSClient.infer_stream()  ──SSE──►  LMStudio :1234    │
 │                                                                         │
 │  5. Parse   StreamProcessor / ContentRouter extract inline tags:        │
 │       [MOOD:happy]  [IMAGE:prompt]  [ACTION:x]  [STAT:health+10] [VOICE] │
 │                                                                         │
 │  6. run_post()  ─ ~7 POST interceptors ─────────────────────────────►   │
 │       75 HeatAwareness · 80 ResponseShaper · 85 TTS · 90 Logger          │
 │       92 MoodSync/SpectatorBroadcast · 93 Relationship                   │
 └─────────────────────────────────────┬───────────────────────────────────┘
                                       ▼
   STATE WRITES + CASCADE
     • CharacterRegistry mood / relationship deltas
     • PlayerState credits / rep / heat   ([STAT:…] tags)
     • ComfyUI image queue                ([IMAGE:…] tags)
     • OutputEvaluator.score → DataCollector  (training flywheel)
     • WorldSim / EventCascade → broadcast to other subscribed scenes
                                       ▼
   Scene emits via Socket.IO:  chat message · HUD update · portrait mood · TTS audio

A few details that make this more than a wrapper:

ResponseContext is a mutable bag passed to every interceptor — system_prompt, messages, reply, auto_results, plus post-call metadata like mood_tags, response_id, and is_stateful. Interceptors read and write it in priority order; any PRE interceptor can abort the LLM call entirely.
Skills have three trigger types (engine/mcp/comms_framework.py): auto (run before the call, result injected), optional (offered to the model as a tool), and required (the model must call it). Auto-skill invocation is throttled by a real COOLDOWN_TRACKER and prerequisite chain (v1.59.0).
Stream tags are the action channel. The LLM doesn’t just talk — it emits [STAT:health+10], [MOOD:x], [IMAGE:prompt] inline, parsed by StreamProcessor (_RE_STAT = \[STAT:(\w+)([+-]\d+)\]) and routed to game state, mood, and the ComfyUI queue.
The cascade is what makes the city feel alive. A WorldSim tick (~60s) emits SimEvents (economy / faction shift / NPC action / weather); EventCascade broadcasts to every subscribed scene, so an NPC’s action in one room ripples through the HUD and faction standings everywhere.

The self-improving loop (why local models punch above their weight) (click for more details)

NEON CITY — the living world & game mechanics

The city breathes. Six factions fight for control. The night never ends. — content/scenes/neoncity/neoncity_scene.py

NEON CITY is CosySim’s flagship: not a single screen but a persistent, autonomous world that the GAME-pillar scenes all plug into. The economy keeps moving, factions keep fighting over turf, NPCs keep walking their daily routines, and the weather keeps rolling through neon-soaked rain — whether or not a player is logged in. When you do walk in, the world has a memory: your last heist raised the heat, your faction standing changes the prices you’re quoted, and a botched job last night is still rippling through three other scenes.

The whole thing runs on local inference (LMStudio). No cloud, no API keys for the simulation itself — a city full of agents reasoning on your own GPU.

One world, many windows

The GAME pillar is a set of independent Flask/Socket.IO scenes (each on its own port, inheriting BaseScene), but they are windows into the same world state , not separate games. NEON CITY (:5563) is the hub; the rest are districts and venues you move through.

Scene	Display name	What it is
`neoncity`	NEON CITY	Living-world hub: economy, factions, missions, crew, cyberspace, board mode
`phone`	SIGNAL	Cyberdeck — messaging, 0xGH0ST contacts, mini-games, the always-on HUD
`penthouse`	THE PENTHOUSE	3D character room (three.js r184), curtain-wall skyline, autonomous NPCs
`heist`	THE HEIST	Crew-driven heist runs that feed heat and faction consequences back into the city
`arena`	THE COLOSSEUM	Combat matches; bookmaking and fighter queues spawned by the world sim
`casino`	CLUB NOIR	Gambling, VIP access gated by faction standing
`lounge`	THE VELVET PIT	Speakeasy social scene, ambient events, brokered deals
`tavern`	THE RUSTY ANCHOR	Fantasy RPG tavern, barter economy, dice
`realm`	LitRPG world	Dual-agent (Director + companion), d20 combat, inventory
`grid`	THE GRID	District board / strategy layer over territory control
`games`	THE ARCADE	Trivia, chess puzzles, leaderboards, AI opponents
…		plus `gallery`, `cyberspace`, and the broader GAME catalogue

The authoritative catalogue lives in engine/control_plane_registry.py (resolved to ports by engine/port_registry.py); the launcher, TUI, and the in-browser THE TERMINAL catalogue all derive their lists from it. See docs/SCENES.md for the full pillar tables.

The living world: two coordinated daemons

NEON CITY’s “aliveness” comes from background daemon threads in engine/world/ that tick on a game clock (1 real second ≈ 1 game minute; 60 real seconds = 1 game hour).

WorldSim (engine/world/world_sim.py) is the event engine. It fires scripted-but-stochastic events on per-task intervals — NPC actions every ~60s, ambient mood shifts, faction shifts, 0xGH0ST hacker messages, arena match queues, major world events, and passive economy ticks. Every SimEvent is (a) appended to a 200-entry ring buffer, (b) persisted to Nexus KMS as world_sim history, and (c) broadcast on the EventBus. A get_digest(scene) call returns “what you missed” the moment you enter a scene.

LivingWorld (engine/world/living_world.py) is the orchestrator. Each tick it coordinates every subsystem in order:

1. Game clock        → read time-of-day from WorldState
2. NPC routines      → RoutineManager moves NPCs to scheduled locations
3. Faction AI        → each faction makes one strategic decision (every 5th tick)
4. Market tick       → supply/demand drift, territory-weighted prices
5. Weather cycle     → Markov transition (CLEAR→NEON_RAIN→STORM→BLACKOUT…)
6. Stochastic events → fire + propagate consequences to market & NPCs

The subsystems, by file (click for more details)

“The city remembers” — the v1.59 / v1.60 feedback loops

Earlier versions simulated a world but the simulation was largely cosmetic — agents could say [STAT:arousal+10], a faction could “expand territory”, and nothing actually changed. The v1.59 “consequential world” and v1.60 “Living Systems” passes closed those loops. This is the part worth studying: it’s where a pile of independent systems became a world with cause and effect.

Loop	Before	After (the closed loop)	Where
Stat tags applied	`[STAT:trust-5]` parsed then discarded	`StatSyncInterceptor` (priority 91) writes tags to real character state before mood rules evaluate them	`engine/agents/interceptors/stat_sync.py`
Economy settlement	Buying/selling was flavour text	`Market._settle_buy/_settle_sell` debit the wallet, move inventory, and raise heat for illegal goods	`engine/world/market.py`
World→market shocks	World events never reached the market	`Market.subscribe_to_world_events()` maps `gang_war→weapons up`, `festival→luxury up`, `shortage→category surge`	`living_world._init_subsystems` → `market.py`
Faction gating	Standing only gated casino VIP	`faction_gates` gives every scene shop access, ally discounts (−10%), rival surcharges (+15%), and standing-locked missions	`engine/world/faction_gates.py`
Player-aware factions	Factions ignored the player	Faction AI weights decisions by your standing — allies expand near you, rivals raid your turf; shifts broadcast on `NEONCITY_FACTION_SHIFT`	`engine/world/faction_ai.py`
Mission consequences	Missions resolved in isolation	`MissionManager._apply_consequences` applies rep/heat/faction-control deltas on success and failure; chains branch on outcome & standing	`mission.py` + `mission_chains.py`
Equipment matters	Equipped gear gave +0	`get_equipment_bonuses` aggregates equipped items into skill/stat deltas that skill-checks consume	`engine/world/equipment_effects.py`
Cross-scene ripples	Events stayed local	`EventCascade` fans `WorldSim` events to subscribed scenes (e.g. a `CRIME` event reaches `phone`, `tavern`, `casino`, `heist`)	`engine/world/event_cascade.py`

The net effect: a heist that goes wrong raises city-wide heat, which HeatAwarenessInterceptor (pipeline priority 75) makes NPCs aware of; the resulting faction shift reprices the black market; and the next mission in the chain unlocks a different branch because your standing changed. Every magic number here is config-driven (mission.consequences.*, economy.event_shocks.*, territory.faction_ai.*) so the whole consequence economy is tunable without touching code.