External Publication
Visit Post

AlphaAvatar v0.6.4: a self-hostable realtime multimodal assistant runtime with shared perception and ENV memory

Hugging Face Forums [Unofficial] July 11, 2026
Source

Hi everyone

A few weeks ago, I shared an early introduction to AlphaAvatar , an open-source and self-hostable realtime full-multimodal personal AI assistant runtime.

The discussion around that post was very helpful. One particularly useful point was that AlphaAvatar may be more valuable when positioned less as another avatar frontend, and more as a reusable runtime layer for persistent realtime multimodal agents.

That is increasingly how I see the project as well.

AlphaAvatar does not try to own every ASR, TTS, RTC, model, or avatar renderer. Instead, it aims to provide the stateful runtime connecting them:

  • realtime multimodal perception

  • session and interaction state

  • persistent Memory and Persona

  • identity continuity

  • tools, MCP, RAG, and DeepResearch

  • status and workflow feedback

  • avatar and channel outputs

  • self-hosted storage and provider control

Since the previous post, the project has gone through several architecture-focused releases, culminating in v0.6.4.

Updated architecture

The new architecture is organized around six main areas:

1. AlphaAvatar Core

avatar-core contains transport-independent multimodal primitives:

  • EnvObservation

  • EnvAnnotation

  • MediaPayload

  • raw and annotated payload views

  • generic video frame buffers

  • typed perception streams

  • shared timelines

  • consumer-specific observation windows

The goal is to let Memory, Persona, Vision, and future runtime modules consume the same multimodal observation model without depending directly on LiveKit or another RTC backend.

2. AlphaAvatar RTC

avatar-rtc acts as the adapter boundary between external realtime communication systems and AlphaAvatar Core.

LiveKit remains the default RTC integration today, but LiveKit-specific frame and track types are converted at the adapter boundary instead of leaking into the core runtime.

The main design principle is:

RTC is an adapter, not business logic.

Future native WebRTC, aiortc, or other realtime backends should be able to publish the same core observations.

3. AlphaAvatar Agent

avatar-agents owns orchestration and session execution.

The runtime is now separated into parallel components:

  • SessionRuntime

  • ContextRuntime

  • PerceptionRuntime

  • IO Runtime

  • Agent Runtime

  • AvatarEngine as the high-level orchestrator

This moves more responsibilities out of AvatarEngine and gives plugins clearer runtime dependencies and lifecycle boundaries.

4. AlphaAvatar Plugins

Cross-cutting assistant capabilities are implemented as plugins:

  • Memory

  • Persona

  • Status

  • Character

  • Interaction Router

  • future Reflection, Planning, and Behavior plugins

Plugins consume runtime context and perception streams instead of independently subscribing to RTC tracks or rebuilding their own session state.

5. Tools Plugins

Tool and knowledge capabilities remain independently replaceable:

  • MCP

  • RAG

  • DeepResearch

  • future sandbox and external workspace integrations

These tools are also connected to Memory and Status so that long-running actions can remain visible and useful across future sessions.

6. Channels

The same assistant runtime can be exposed through:

  • the realtime web application

  • WhatsApp

  • avatar interfaces

  • future Discord, Slack, mobile, and external application integrations

The channel should change how the assistant is accessed, not recreate the assistant itself.

Shared realtime perception

One of the main changes in v0.6.4 is the introduction of a shared perception path:

RTC / Device Input
        ↓
EnvObservation + MediaPayload
        ↓
PerceptionRuntime
  ├── Typed Streams
  ├── Shared Timeline
  └── Window Builder
        ↓
Persona / Vision / Memory / Future Routers

Previously, different modules could subscribe to the LiveKit video track independently.

Now, an RTC adapter publishes the media once. Multiple runtime consumers can process it using their own cursors, sampling policies, buffers, and temporal windows.

For example:

  • Persona can perform face detection and identity matching.

  • Vision can select recent frames for the current LLM turn.

  • Memory can consume a longer observation window.

  • Future routers can use motion, presence, screen, or speaker events.

These consumers do not have to block one another or use the same sampling rate.

Online ENV Memory

v0.6.4 also introduces the first version of online ENV Memory.

Traditional assistant memory usually begins with conversation text:

user message
    ↓
memory extraction
    ↓
long-term memory

ENV Memory extends this to continuously observed multimodal context:

live visual observations
        ↓
ordered perception window
        ↓
multimodal ENV extraction
        ↓
structured environment memory
        ↓
future visual-history retrieval

Examples of ENV memories include:

  • a person remaining at a desk during a session

  • a cup being placed on a table

  • a plant repeatedly appearing near the user

  • a screen showing a coding workflow

  • a person entering or leaving the visible environment

  • an object changing position across observation windows

The goal is not to store every frame or generate endless image captions.

Instead, AlphaAvatar samples observations, aligns annotations, extracts concise episodic memories, and stores only useful structured results.

Raw video frames remain runtime-only and are not written directly into the vector database or Markdown memory storage.

Annotations as alternate payload views

Persona can now attach face annotations to shared observations.

An observation may contain multiple representations:

  • raw video frame

  • raw JPEG

  • annotated video frame

  • annotated JPEG

  • future derived representations

Memory can prefer an annotated JPEG when extracting ENV context, while Vision can resolve the most recent annotated frame when building model context.

The raw representation remains unchanged.

This allows different modules to enrich the same observation without copying or overwriting the original media.

Graph-aware multimodal memory

The previous v0.6.3 release introduced graph-aware Memory foundations:

  • multi-object memory ownership

  • graph node mentions

  • session-scoped local entity keys

  • alias-ready graph lookup

  • LanceDB graph-node indexing

  • graph-aware semantic retrieval

ENV Memory now plugs into the same system.

A visual memory may be connected to concrete observed entities such as:

  • a visible person

  • a face track

  • a screen

  • a document

  • a plant

  • a vehicle

  • a physical object

  • a distinguishable room

The memory text remains the source of truth, while graph nodes act as sparse retrieval anchors.

Provider-independent extraction

Another recent change is the task-based Provider Layer.

Memory, Persona, embeddings, and multimodal extraction tasks can be configured independently.

For example:

  • one model may extract conversation memory

  • another multimodal model may process ENV observations

  • a local embedding model may index memories

  • another provider may handle the realtime assistant response

The personal data and storage layer can remain self-hosted even when optional external inference providers are used.

Why keep this outside the model?

It is possible that more orchestration will eventually become model-native.

Models may increasingly handle:

  • context compression

  • tool planning

  • multimodal state tracking

  • memory selection

  • user modeling

However, I still think an explicit user-owned runtime remains valuable for:

  • privacy boundaries

  • persistent personal storage

  • model and provider swapping

  • observability

  • replay and evaluation

  • identity management

  • tool permissions

  • debugging

  • user-editable Memory and Persona

  • cross-channel continuity

The model can become more capable without requiring personal state, tools, and system behavior to disappear into an opaque kernel.

Current limitations

AlphaAvatar is still developer-first and under active development.

Current limitations include:

  • LiveKit is still the primary RTC implementation.

  • ENV Memory currently focuses mainly on sampled visual observations.

  • Visual-history retrieval is still relatively basic.

  • Multi-user visual and speaker routing requires more work.

  • Reflection, Planning, Behavior, and Interaction Router are not complete.

  • User-facing Memory and Persona inspection controls are still being developed.

Next steps

The next major directions include:

  • audio and screen-based ENV Memory

  • richer object-, event-, identity-, and time-aware retrieval

  • cross-window event consolidation

  • multi-user face and speaker alignment

  • alternative RTC adapters

  • interaction routing and proactive behavior

  • Reflection and Planning

  • replay and evaluation tooling

  • user-controlled Memory and Persona editing

Questions for the community

I would be interested in feedback on a few design questions:

  1. Which assistant capabilities should remain explicit runtime components, and which should eventually become model-native?

  2. What is the right evaluation framework for long-running ENV Memory?

  3. Should multimodal memory primarily store textual episodes, structured events, graph relations, selected media artifacts, or a combination?

  4. What runtime interface would make AlphaAvatar useful to other voice-agent, digital-human, AI companion, or AI VTuber projects?

Links

  • GitHub

  • Previous Hugging Face discussion

Thanks to everyone who provided feedback on the original post. The architecture is still evolving, and contributions, criticism, comparisons with related projects, and implementation discussions are very welcome.

Discussion in the ATmosphere

Loading comments...