{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreidugr6xdbsx4g5henuqmebbet3mgf2bu4rst6npgf6tgqwml2kbba",
    "uri": "at://did:plc:3pjw65epwlo3rzajhx6xg4br/app.bsky.feed.post/3mpgbqwx2vpb2"
  },
  "path": "/2026/06/29/2026-06-28-building-local-offline-opensuse-assistant/",
  "publishedAt": "2026-06-29T08:00:21.000Z",
  "site": "https://news.opensuse.org",
  "tags": [
    "Source repository",
    "GSoC proposal",
    "Public Leap VM demo",
    "OBS package",
    "evaluation report",
    "Rudraksh Karpe",
    "Satyam Soni"
  ],
  "textContent": "We started this Google Summer of Code project with a simple question: can a new openSUSE user get useful, system-specific help without sending their questions or machine information to a cloud service?\n\nThe idea was to combine a Small Language Model (SLM) running on the user’s machine with retrieval over official openSUSE documentation. Instead of giving generic Linux advice, the assistant should know that the machine is running Leap, whether it uses Btrfs and Snapper, which GPU is installed, and which services have failed. It should then explain tools such as `zypper`, YaST, Snapper, and `firewalld` using the documentation that applies to that system.\n\nAt the midterm point, that core loop is working on an openSUSE Leap 16.0 VM. The assistant runs locally, retrieves openSUSE documentation from a local LanceDB index, reads selected host context, and is available through both a command-line interface and a web UI. More importantly, it now runs from an openSUSE BCI-based container built by the Open Build Service (OBS), including on a VM with no outbound internet access.\n\n  * Source repository\n  * GSoC proposal\n  * Public Leap VM demo\n  * OBS package\n\n\n\n## From a Small PoC to a Real Leap Deployment\n\nBefore GSoC, the project used TinyLlama, ChromaDB, and a small set of documentation to prove that the basic architecture was possible. That was enough for a demo, but not enough to answer the questions that matter for a distribution feature: Which model works well on CPU-only hardware? How is the assistant installed? What happens when the target machine cannot download a model? Can a container inspect the host without being given unnecessary privileges?\n\nThe mentor-provided test VM made these questions concrete. It runs Leap 16.0 with 4 vCPUs and 15 GB of RAM, has no GPU, and cannot access the internet. This is a useful target because it prevents an accidental dependency on cloud APIs or a developer’s already-populated cache.\n\nThe current request path is:\n\n  1. The user asks a question in the CLI or Gradio web UI.\n  2. The assistant reads relevant, non-sensitive system facts from the host.\n  3. MiniLM embeddings retrieve matching chunks from a local LanceDB index.\n  4. The system prompt combines the question, host context, and retrieved sources.\n  5. A local GGUF model generates the answer through `llama.cpp`.\n  6. The answer includes references to the documentation used.\n\n\n\nThe index currently contains 1,982 chunks from the Leap Startup and Reference guides, Leap 16.0 release notes, and selected openSUSE wiki SDB pages. The SDB set includes common areas such as repositories, upgrades, audio, and NVIDIA driver and SUSE Prime troubleshooting.\n\n## Model Selection Needed Measurement, Not Guesswork\n\nOne of the first tasks was replacing TinyLlama with models that could produce reliable administration guidance while still running on ordinary hardware. We added model tiers and benchmarked six GGUF models through the complete RAG and system-context pipeline on the Leap VM.\n\nEach model answered the same eight openSUSE onboarding questions. Generation ran offline on the VM, while judging ran separately against saved answers using Gemini 2.5 Flash-Lite, reference answers, and expected-fact checklists. Separating these steps meant the slow CPU generation did not need to be repeated when the scoring method changed.\n\nThe benchmark used the same constrained target as the public deployment: 4 vCPUs, 15 GB of RAM, no GPU, and no network access during generation. Latency below is the average time for one complete answer, including retrieval and the prompt built from the retrieved documentation and host context.\n\n_Higher and further left is better. Qwen3-4B achieved the highest judged quality; Gemma 4 E4B is the current default and answered about 30 seconds faster. The orange ring marks the current default._\n\nModel | Quality (1-5) | Average latency\n---|---|---\nQwen3-4B-Instruct | 4.88 | 106 s\nQwen3-8B | 4.75 | 136 s\nGemma 4 E4B | 4.62 | 76 s\nGemma 3 4B | 4.50 | 77 s\nQwen3-1.7B | 4.00 | 42 s\nTinyLlama 1.1B | 2.62 | 12 s\n\nThe results showed that a larger model is not automatically a better default. Qwen3-4B scored slightly higher than Qwen3-8B and answered about 30 seconds faster. Gemma 4 E4B offered a useful latency/quality trade-off and is currently the default standard model following mentor discussion. Qwen3-1.7B remains a practical option for lower-resource machines, while TinyLlama is useful only for smoke tests.\n\nThe full methodology and per-model discussion are in the evaluation report.\n\nThese timings also changed the web UI. A 70-120 second answer can look like a hung application if the interface says nothing. The UI now identifies the selected tier, explains that the model is loaded on the first question, shows elapsed generation time, and reports missing model or index data more clearly.\n\n## Offline Means More Than Running the Model Locally\n\nThe largest lesson so far is that local inference and offline installation are two different problems.\n\nOnce downloaded, a GGUF model can run without a network connection. A usable assistant, however, also needs the embedding model, Python and native libraries, the documentation index, configuration, and compatible file ownership. The VM’s lack of internet access exposed every hidden download quickly.\n\nTo make the state visible, we added `suse-assist doctor`. It checks model files, the vector store, the embedding cache, memory and disk availability, host mounts, offline environment settings, the web port, and the container runtime. We also added `suse-assist setup` to select a tier, prepare the model and index, optionally import offline data, and run the final checks.\n\nFor machines that cannot download assets during setup, the project now has an offline bundle format:\n\n\n    suse-assist bundle export --output suse-assist-offline-bundle.tar.zst\n    suse-assist bundle import suse-assist-offline-bundle.tar.zst\n\n\nA bundle contains a selected GGUF model, the MiniLM cache, the LanceDB index, and a manifest with checksums. The import/export commands and firstboot import path are implemented. Producing a release bundle from the VM data and testing it through a complete OEM first boot is one of the next tasks. The final distribution policy is still open: the data could be part of a KIWI image, an RPM payload, a separate artifact, or a firstboot download for connected systems.\n\n## Moving the Build into the openSUSE Infrastructure\n\nThe first container image was useful for testing, but it was not built in the same way an openSUSE deliverable should be. The container now starts from the SUSE BCI Python base and uses `zypper` for system dependencies.\n\nThe next issue was dependency availability. OBS builds in a clean, offline build environment, while several parts of the Python ML stack are not currently packaged as openSUSE RPMs. This includes important pieces around `llama-cpp-python` and LanceDB. To test the rest of the path without waiting for all dependency packaging, we built an OBS prototype with an approximately 467 MB wheelhouse. It installs the Python dependencies with `pip --no-index`, including a locally built `llama-cpp-python` wheel.\n\nThat experiment now builds the real assistant image in `home:anujagrawal:suse-assist/suse-assist-image` and publishes it as:\n\n\n    registry.opensuse.org/home/anujagrawal/suse-assist/images/opensuse/suse-assist:latest\n\n\nThe Leap VM cannot pull from the registry, so we tested the actual offline delivery path: download the OBS image archive on a connected machine, checksum it, copy it over SSH, load it with Podman, and run it against the prepared data volume. This found a real ownership mismatch, which was fixed by assigning the container’s `suseai` user a stable UID and GID of 999. The public demo now runs this OBS-built image rather than the earlier local build.\n\nOn that VM, `suse-assist doctor`, web startup, test-tier inference, and standard Gemma 4 E4B inference have all been exercised. A standard-tier answer to “What is zypper?” took about 70 seconds and included retrieved references.\n\nThe vendored wheelhouse proves that OBS can build and publish the container. It is not automatically the final packaging answer. The open question is whether this is acceptable as a short-term container solution or whether each missing dependency should be packaged as a proper RPM before the assistant moves beyond a home project.\n\n## Preparing for First Boot and Daily Use\n\nDistribution integration should not require users to know a container command. The repository now includes the first scaffolding for a KIWI OEM image, a Podman Quadlet unit, and a firstboot service that prepares the assistant’s data and starts the web service. There is also a native systemd service setup path for a future RPM install, plus a desktop launcher that opens the local UI.\n\nThese pieces define how the assistant could be presented as a normal openSUSE feature, but they are not the same as a completed installer integration. The full KIWI image and offline firstboot flow still need end-to-end validation. Integration with Agama or `jeos-firstboot` also needs to follow the packaging and data distribution decisions rather than hard-code a temporary deployment method.\n\nWe also added an MCP server and client proof of concept. The server exposes system context and documentation search as tools, while the client can connect the assistant to external MCP servers. This is not required for the basic onboarding flow, but it shows that the openSUSE-specific context and retrieval work can be reused by other local applications.\n\n## Safety Is Part of the Interface\n\nSystem administration answers have more impact than ordinary chatbot responses. Retrieved web pages are therefore treated as untrusted context: instructions inside the retrieved text must not override the system prompt. The assistant cleans hidden reasoning markers, asks for documentation references when recommending commands, and adds warnings around destructive operations and repository vendor changes.\n\nThe container runtime also uses a non-root user, dropped capabilities, a read-only root filesystem where possible, resource limits, and a health check. Host system context is mounted read-only. These controls do not make generated advice infallible, but they reduce the authority of the process and make risky advice more visible to the user.\n\n## What Changed from the Proposal\n\nThe original midterm plan placed more emphasis on completing a `jeos-firstboot` module and systemd socket activation by this point. The core assistant, model tiers, RAG expansion, system-context work, and Agama integration research are in place, but the VM changed the order of the remaining work.\n\nAn onboarding assistant that works only after a developer manually populates model and embedding caches is not ready for first boot. We therefore moved some packaging and offline-distribution work forward: the BCI migration, OBS build, registry publication, setup and doctor commands, bundle format, and offline VM validation. The firstboot integration is now being built on top of a deployment path that has actually run on Leap without internet access.\n\nThis was a useful correction to the plan. The main difficulty is not writing one more chat interface. It is delivering several gigabytes of model data and a native ML dependency stack through normal distribution infrastructure, then making the result understandable on machines with very different resources.\n\n## Plans for the Second Half\n\nThe next part of the project will focus on closing that delivery gap:\n\n  1. Build a real offline data bundle from the VM and test bundle import through the OEM firstboot path.\n  2. Settle the short-term OBS dependency approach: vendored wheels or proper RPMs for the missing ML stack.\n  3. Produce and validate an installable RPM on a clean Leap system.\n  4. Decide how model weights and the generated documentation index are distributed, including licensing and image-size implications.\n  5. Complete the KIWI/firstboot path and connect it to the appropriate openSUSE onboarding surface.\n  6. Continue expanding evaluation coverage for common problems such as codecs, Packman vendor changes, Wi-Fi and Bluetooth, Btrfs snapshot disk usage, and failed systemd services.\n\n\n\nOBS images from home projects are already published to `registry.opensuse.org`, so development can continue in the current home project. An appropriate long-term development project, potentially an AI-assistant or AI-containers namespace, is being discussed with the openSUSE infrastructure team.\n\nThe project has reached the point where the assistant itself works. The second half is about making it an openSUSE feature that users can install, start, diagnose, and use offline without knowing how its model and retrieval stack are assembled.\n\nThanks to my mentors, Rudraksh Karpe and Satyam Soni, and to the openSUSE community members who have provided the Leap VM, documentation pointers, packaging guidance, and feedback on the live demo.",
  "title": "We started this Google Summer of Code project with a simple question: can a new openSUSE user get useful, system-specific help without sending their questions or machine information to a cloud service?"
}