Sovereign AI Stack — Architecture

Self-hosted, air-gappable chat with live web search via Searxng. No external API calls, no cloud LLM dependency. Live at chat.lab.local.

The whole stack runs on lab hardware in Cyberjaya. Two K3s nodes (mailt03u for control + mailgb02 for GPU inference) plus the user's browser. User queries → keyword-aware shim → live Searxng metasearch → Gemma 4 26B-A4B → cited answer. No request ever leaves the lab network except the Searxng → public search engines hop, and that is anonymous.

Request flow (the path of one chat message)

   ┌──────────────────────────────────────────────────────────────────────────┐
   │  Browser  (zahir on Tailscale, or anyone on the lab LAN)                 │
   └────────────────────────────────┬─────────────────────────────────────────┘
                                    │ HTTPS  chat.lab.local
                                    ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │  LibreChat  (ns librechat · MongoDB-backed · Keycloak SSO)               │
   │  • model picker · conversation history · Dell branding                   │
   └────────────────────────────────┬─────────────────────────────────────────┘
                                    │ HTTP  POST /v1/chat/completions
                                    ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │  strip-shim   (ns vllm · svc: litellm · ~120 lines Python aiohttp)       │
   │                                                                          │
   │  1. Strip junk fields LibreChat sends but vLLM rejects                   │
   │     • tools: [] (empty array — rejected as 400 by vLLM)                  │
   │     • web_search_options (LibreChat-only)                                │
   │                                                                          │
   │  2. Scan the latest user message for a TRIGGER:                          │
   │     • EN: today, latest, news, weather, near me, http://...              │
   │     • MS: hari ini, terkini, berita, cuaca, dekat sini                   │
   │                                                                          │
   │  3. If triggered, query Searxng for the user's text, take top 5 results, │
   │     prepend as a system message, and TRIM history so old "I cannot       │
   │     access the internet" replies stop anchoring the answer.              │
   └────────────────────────────────┬─────────────────────────────────────────┘
                                    │
                          ┌─────────┴────────┐
                          │ no trigger       │ triggered
                          │                  ▼
                          │      ┌──────────────────────────────────┐
                          │      │  Searxng (ns searxng)             │
                          │      │  Anonymous metasearch over        │
                          │      │  Google · Bing · DuckDuckGo ·     │
                          │      │  Brave · Wikipedia                │
                          │      │  → top 5 JSON results back        │
                          │      └──────────────────────────────────┘
                          ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │  litellm-direct   (ns vllm · the real LiteLLM proxy)                     │
   │  • routes by model name · master_key auth · drop_params                  │
   └────────────────────────────────┬─────────────────────────────────────────┘
                                    │
                  ┌─────────────────┼─────────────────────┐
                  ▼                 ▼                     ▼
   ┌──────────────────┐ ┌──────────────────┐ ┌────────────────────────┐
   │ vllm-gemma4-26b  │ │ vllm-gemma3-12b  │ │ vllm-qwen3vl-32b       │
   │ • MoE, 4B active │ │ • dense          │ │ • vision specialist    │
   │ • 32 t/s         │ │ • 15 t/s         │ │ • 6.5 t/s              │
   │ • DEFAULT        │ │ • text only      │ │ • image Q&A demos      │
   │ • Tool-call OK   │ │ • no tools       │ │ • hermes parser        │
   └──────────────────┘ └──────────────────┘ └────────────────────────┘
        all three on mailgb02 (192.168.1.9) · NVIDIA GB10 · 119.6 GB GPU

Why a custom shim, not LibreChat agents

LibreChat's Agents feature looks like the right tool — there is even a Web Search tool in the UI — but the agent path uses vLLM's /v1/responses endpoint, and that endpoint does not run vLLM's tool-call parsers. We confirmed this on 2026-05-28: an agent backed by Gemma 4 (which has a working gemma4 parser on /v1/chat/completions) returned its native <|tool_call>...<tool_call|> markup as plain text. LibreChat saw no structured tool call, called nothing, and the user got an empty bubble.

So instead of fighting that, the shim does the work transparently for the normal Custom-endpoint chat path. No agent UI, no surprises, and it works with any model in the lab — Gemma 3, Gemma 4, Qwen3-VL — none of them need to support tool calling.

Components

Component	Namespace	Role	Resource cost
LibreChat	`librechat`	Chat UI · Mongo-backed history · Keycloak OIDC · model picker	~1 vCPU, ~1 GB RAM
strip-shim NEW	`vllm`	Strips empty-tools junk · keyword-triggered Searxng auto-search · trims history	~50 MB RAM, no GPU
Searxng NEW	`searxng`	Anonymous metasearch over Google / Bing / DuckDuckGo / Brave / Wikipedia	~150 MB RAM, no GPU
LiteLLM	`vllm`	OpenAI-compatible proxy · model routing · master key · param dropping	~200 MB RAM
vLLM × 3	`vllm`	The actual LLM inference engines · FP8 on GB10	~75 GB unified GPU memory total

What the shim actually does (annotated)

# 1. Junk-stripping (always)
if data.get("tools") == []:
    data.pop("tools", None)
    data.pop("tool_choice", None)
data.pop("web_search_options", None)

# 2. Trigger detection (regex over English + Bahasa Malaysia)
TRIGGERS = r"\b(today|tonight|latest|news|near (me|here)|hari ini|terkini|berita|cuaca|...)\b|https?://\S+"

# 3. If triggered:
results_md = searxng(query=user_text, k=5)   # top-5 JSON results
system_msg = SYSTEM_PROMPT.format(query=user_text, results=results_md)

# 4. Trim history so prior "I can't access the internet" replies
#    don't anchor the model's behaviour
data["messages"] = [system_msg, latest_user_msg]

Trigger keywords (current list)

🇬🇧 English

today tonight yesterday tomorrow now current latest recent breaking news weather stock price score live update near me nearby this week / month / year

🇲🇾 Bahasa Malaysia / Indonesia

hari ini semalam esok terkini baru-baru berita cuaca harga dekat sini berdekatan

🔗 URLs

Any message containing http:// or https:// triggers a search for that URL's context.

Example: what happens for "best nasi lemak in Shah Alam tonight"

LibreChat sends POST /v1/chat/completions with the user message and a stale tools: []
Shim drops the empty tools[] array (vLLM would have returned 400 otherwise)
Trigger matches: tonight
Shim queries Searxng → gets 5 results from Yelp, Tripadvisor, TikTok, etc.
Shim prepends a system message: "LIVE WEB SEARCH RESULTS ... CRITICAL: never say you cannot access the internet"
Shim trims history to just [system_with_results, latest_user_msg]
Request flows to LiteLLM → Gemma 4 26B-A4B on mailgb02
Gemma 4 answers in ~5 s, citing "according to Yelp", "noted on TikTok", etc.

What is not here

This is web search, not RAG. Lab-specific questions ("which projects use the GB10?", "what is the Coolest Path API?") still rely on the model's training data plus whatever Searxng can find publicly. Building a pgvector + embeddings layer over innovation.lab.local, ai-wiki.lab.local, and the Forgejo READMEs is the next step — see the rag-plan in memory for the design.

Sovereignty checklist

Concern	Status	Notes
LLM weights	Self-hosted	Gemma 3 / Gemma 4 / Qwen3-VL · FP8 quantised · loaded into GB10 memory
Chat UI	Self-hosted	LibreChat in K3s · MongoDB local
Search index	Federated	Searxng is self-hosted, but it federates queries out to public engines anonymously
Auth	Self-hosted	Keycloak OIDC, lab-internal realm `ailab`
TLS	Self-hosted	cert-manager · lab CA · *.lab.local wildcard
DNS	Self-hosted	dnsmasq on mailt01 + CoreDNS in cluster
Outbound exposure	Searxng → upstreams	Only Searxng container reaches the public internet; users' queries are sanitised and round-robined across upstreams

Try it

Go to chat.lab.local · log in with Keycloak · pick model gemma4-26b · ask one of these and watch the source citations appear:

Best nasi lemak near Seksyen 13 Shah Alam tonight
Cuaca hari ini di Cyberjaya
What is the latest news about Malaysia today?
NVIDIA stock price today

And for general-knowledge questions where no trigger word fires, the model just answers from its weights — no unnecessary search:

Explain what a GPU cluster is
Write Python to parse JSON