{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreihw4b2npttkf5i2bjzd73cjtv37kmczpgdkvyvaqd5am6plywumfe",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mh2ycc4bnhu2"
  },
  "path": "/t/what-is-the-best-architecture-for-integrating-local-llm-inference-and-rag-on-mobile-devices/174270#post_1",
  "publishedAt": "2026-03-15T02:31:11.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "Hi everyone,\n\nI’m currently exploring a mobile AI architecture and would love to hear technical opinions from others working in this area.\n\nThe goal is to support the following on a mobile app:\n\n  * on-device LLM inference\n\n  * local or hybrid RAG retrieval\n\n  * low-latency interaction\n\n  * integration with Flutter or another cross-platform frontend\n\n\n\n\nThe technical directions I’m considering include:\n\n  * Flutter / cross-platform frontend\n\n  * llama.cpp or another on-device LLM runtime\n\n  * vector retrieval or a lightweight local knowledge base\n\n  * Platform Channel, FFI, or another native bridging approach\n\n\n\n\nMy main questions are:\n\n  1. What is currently the most reliable architecture for **local LLM + RAG on mobile**?\n\n  2. If the frontend is Flutter, would you recommend **Platform Channels** or **FFI**?\n\n  3. What are good approaches for **local knowledge retrieval** on mobile devices?\n\n  4. How do you usually balance performance, memory usage, and model size in production or prototype setups?\n\n\n\n\nI’d be very interested in hearing any real-world experience or recommendations related to mobile AI / edge AI systems.",
  "title": "What is the best architecture for integrating local LLM inference and RAG on mobile devices?"
}