{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreigeoladrqbu5vnfjyezyncixz5232v6czai2jtptv7cae4rdmvo2y",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mqekmf6lhuz2"
  },
  "path": "/t/alphaavatar-v0-6-4-a-self-hostable-realtime-multimodal-assistant-runtime-with-shared-perception-and-env-memory/177684#post_1",
  "publishedAt": "2026-07-11T10:42:14.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "GitHub",
    "Previous Hugging Face discussion"
  ],
  "textContent": "Hi everyone\n\nA few weeks ago, I shared an early introduction to **AlphaAvatar** , an open-source and self-hostable realtime full-multimodal personal AI assistant runtime.\n\nThe discussion around that post was very helpful. One particularly useful point was that AlphaAvatar may be more valuable when positioned less as another avatar frontend, and more as a reusable runtime layer for persistent realtime multimodal agents.\n\nThat is increasingly how I see the project as well.\n\nAlphaAvatar does not try to own every ASR, TTS, RTC, model, or avatar renderer. Instead, it aims to provide the stateful runtime connecting them:\n\n  * realtime multimodal perception\n\n  * session and interaction state\n\n  * persistent Memory and Persona\n\n  * identity continuity\n\n  * tools, MCP, RAG, and DeepResearch\n\n  * status and workflow feedback\n\n  * avatar and channel outputs\n\n  * self-hosted storage and provider control\n\n\n\n\nSince the previous post, the project has gone through several architecture-focused releases, culminating in **v0.6.4**.\n\n## Updated architecture\n\nThe new architecture is organized around six main areas:\n\n### 1. AlphaAvatar Core\n\n`avatar-core` contains transport-independent multimodal primitives:\n\n  * `EnvObservation`\n\n  * `EnvAnnotation`\n\n  * `MediaPayload`\n\n  * raw and annotated payload views\n\n  * generic video frame buffers\n\n  * typed perception streams\n\n  * shared timelines\n\n  * consumer-specific observation windows\n\n\n\n\nThe goal is to let Memory, Persona, Vision, and future runtime modules consume the same multimodal observation model without depending directly on LiveKit or another RTC backend.\n\n### 2. AlphaAvatar RTC\n\n`avatar-rtc` acts as the adapter boundary between external realtime communication systems and AlphaAvatar Core.\n\nLiveKit remains the default RTC integration today, but LiveKit-specific frame and track types are converted at the adapter boundary instead of leaking into the core runtime.\n\nThe main design principle is:\n\n> RTC is an adapter, not business logic.\n\nFuture native WebRTC, `aiortc`, or other realtime backends should be able to publish the same core observations.\n\n### 3. AlphaAvatar Agent\n\n`avatar-agents` owns orchestration and session execution.\n\nThe runtime is now separated into parallel components:\n\n  * `SessionRuntime`\n\n  * `ContextRuntime`\n\n  * `PerceptionRuntime`\n\n  * IO Runtime\n\n  * Agent Runtime\n\n  * `AvatarEngine` as the high-level orchestrator\n\n\n\n\nThis moves more responsibilities out of `AvatarEngine` and gives plugins clearer runtime dependencies and lifecycle boundaries.\n\n### 4. AlphaAvatar Plugins\n\nCross-cutting assistant capabilities are implemented as plugins:\n\n  * Memory\n\n  * Persona\n\n  * Status\n\n  * Character\n\n  * Interaction Router\n\n  * future Reflection, Planning, and Behavior plugins\n\n\n\n\nPlugins consume runtime context and perception streams instead of independently subscribing to RTC tracks or rebuilding their own session state.\n\n### 5. Tools Plugins\n\nTool and knowledge capabilities remain independently replaceable:\n\n  * MCP\n\n  * RAG\n\n  * DeepResearch\n\n  * future sandbox and external workspace integrations\n\n\n\n\nThese tools are also connected to Memory and Status so that long-running actions can remain visible and useful across future sessions.\n\n### 6. Channels\n\nThe same assistant runtime can be exposed through:\n\n  * the realtime web application\n\n  * WhatsApp\n\n  * avatar interfaces\n\n  * future Discord, Slack, mobile, and external application integrations\n\n\n\n\nThe channel should change how the assistant is accessed, not recreate the assistant itself.\n\n## Shared realtime perception\n\nOne of the main changes in v0.6.4 is the introduction of a shared perception path:\n\n\n    RTC / Device Input\n            ↓\n    EnvObservation + MediaPayload\n            ↓\n    PerceptionRuntime\n      ├── Typed Streams\n      ├── Shared Timeline\n      └── Window Builder\n            ↓\n    Persona / Vision / Memory / Future Routers\n\n\n\nPreviously, different modules could subscribe to the LiveKit video track independently.\n\nNow, an RTC adapter publishes the media once. Multiple runtime consumers can process it using their own cursors, sampling policies, buffers, and temporal windows.\n\nFor example:\n\n  * Persona can perform face detection and identity matching.\n\n  * Vision can select recent frames for the current LLM turn.\n\n  * Memory can consume a longer observation window.\n\n  * Future routers can use motion, presence, screen, or speaker events.\n\n\n\n\nThese consumers do not have to block one another or use the same sampling rate.\n\n## Online ENV Memory\n\nv0.6.4 also introduces the first version of **online ENV Memory**.\n\nTraditional assistant memory usually begins with conversation text:\n\n\n    user message\n        ↓\n    memory extraction\n        ↓\n    long-term memory\n\n\n\nENV Memory extends this to continuously observed multimodal context:\n\n\n    live visual observations\n            ↓\n    ordered perception window\n            ↓\n    multimodal ENV extraction\n            ↓\n    structured environment memory\n            ↓\n    future visual-history retrieval\n\n\n\nExamples of ENV memories include:\n\n  * a person remaining at a desk during a session\n\n  * a cup being placed on a table\n\n  * a plant repeatedly appearing near the user\n\n  * a screen showing a coding workflow\n\n  * a person entering or leaving the visible environment\n\n  * an object changing position across observation windows\n\n\n\n\nThe goal is not to store every frame or generate endless image captions.\n\nInstead, AlphaAvatar samples observations, aligns annotations, extracts concise episodic memories, and stores only useful structured results.\n\nRaw video frames remain runtime-only and are not written directly into the vector database or Markdown memory storage.\n\n## Annotations as alternate payload views\n\nPersona can now attach face annotations to shared observations.\n\nAn observation may contain multiple representations:\n\n  * raw video frame\n\n  * raw JPEG\n\n  * annotated video frame\n\n  * annotated JPEG\n\n  * future derived representations\n\n\n\n\nMemory can prefer an annotated JPEG when extracting ENV context, while Vision can resolve the most recent annotated frame when building model context.\n\nThe raw representation remains unchanged.\n\nThis allows different modules to enrich the same observation without copying or overwriting the original media.\n\n## Graph-aware multimodal memory\n\nThe previous v0.6.3 release introduced graph-aware Memory foundations:\n\n  * multi-object memory ownership\n\n  * graph node mentions\n\n  * session-scoped local entity keys\n\n  * alias-ready graph lookup\n\n  * LanceDB graph-node indexing\n\n  * graph-aware semantic retrieval\n\n\n\n\nENV Memory now plugs into the same system.\n\nA visual memory may be connected to concrete observed entities such as:\n\n  * a visible person\n\n  * a face track\n\n  * a screen\n\n  * a document\n\n  * a plant\n\n  * a vehicle\n\n  * a physical object\n\n  * a distinguishable room\n\n\n\n\nThe memory text remains the source of truth, while graph nodes act as sparse retrieval anchors.\n\n## Provider-independent extraction\n\nAnother recent change is the task-based Provider Layer.\n\nMemory, Persona, embeddings, and multimodal extraction tasks can be configured independently.\n\nFor example:\n\n  * one model may extract conversation memory\n\n  * another multimodal model may process ENV observations\n\n  * a local embedding model may index memories\n\n  * another provider may handle the realtime assistant response\n\n\n\n\nThe personal data and storage layer can remain self-hosted even when optional external inference providers are used.\n\n## Why keep this outside the model?\n\nIt is possible that more orchestration will eventually become model-native.\n\nModels may increasingly handle:\n\n  * context compression\n\n  * tool planning\n\n  * multimodal state tracking\n\n  * memory selection\n\n  * user modeling\n\n\n\n\nHowever, I still think an explicit user-owned runtime remains valuable for:\n\n  * privacy boundaries\n\n  * persistent personal storage\n\n  * model and provider swapping\n\n  * observability\n\n  * replay and evaluation\n\n  * identity management\n\n  * tool permissions\n\n  * debugging\n\n  * user-editable Memory and Persona\n\n  * cross-channel continuity\n\n\n\n\nThe model can become more capable without requiring personal state, tools, and system behavior to disappear into an opaque kernel.\n\n## Current limitations\n\nAlphaAvatar is still developer-first and under active development.\n\nCurrent limitations include:\n\n  * LiveKit is still the primary RTC implementation.\n\n  * ENV Memory currently focuses mainly on sampled visual observations.\n\n  * Visual-history retrieval is still relatively basic.\n\n  * Multi-user visual and speaker routing requires more work.\n\n  * Reflection, Planning, Behavior, and Interaction Router are not complete.\n\n  * User-facing Memory and Persona inspection controls are still being developed.\n\n\n\n\n## Next steps\n\nThe next major directions include:\n\n  * audio and screen-based ENV Memory\n\n  * richer object-, event-, identity-, and time-aware retrieval\n\n  * cross-window event consolidation\n\n  * multi-user face and speaker alignment\n\n  * alternative RTC adapters\n\n  * interaction routing and proactive behavior\n\n  * Reflection and Planning\n\n  * replay and evaluation tooling\n\n  * user-controlled Memory and Persona editing\n\n\n\n\n## Questions for the community\n\nI would be interested in feedback on a few design questions:\n\n  1. Which assistant capabilities should remain explicit runtime components, and which should eventually become model-native?\n\n  2. What is the right evaluation framework for long-running ENV Memory?\n\n  3. Should multimodal memory primarily store textual episodes, structured events, graph relations, selected media artifacts, or a combination?\n\n  4. What runtime interface would make AlphaAvatar useful to other voice-agent, digital-human, AI companion, or AI VTuber projects?\n\n\n\n\n## Links\n\n  * GitHub\n\n  * Previous Hugging Face discussion\n\n\n\n\nThanks to everyone who provided feedback on the original post. The architecture is still evolving, and contributions, criticism, comparisons with related projects, and implementation discussions are very welcome.",
  "title": "AlphaAvatar v0.6.4: a self-hostable realtime multimodal assistant runtime with shared perception and ENV memory"
}