What is a local LLM brain architecture?

A local LLM brain architecture is a self-hosted AI assistant system that combines local models, routing, memory, tools, privacy rules, and observability instead of relying on one model or one chat UI.

Why move from OpenClaw to Hermes?

OpenClaw helped frame the first self-hosted assistant, but Hermes became the better live orchestrator because it owned the chat channels, cron jobs, routing, and local second-brain workflow in one place.

What hardware do you need for a self-hosted AI assistant?

You can start with one capable machine, but the architecture scales better when consumer hardware has assigned roles: a primary host, a heavy reasoning tier, a vision-capable node, and a reliable fallback.

Is a personal AI brain just one large local model?

No. The durable architecture is a system: routing, model roles, memory discipline, privacy boundaries, monitoring, and governance. A larger model helps only when the rest of the system can use it reliably.

← Back to all posts Local LLM brain architecture evolution from OpenClaw to Hermes with routing and memory layers

Local LLM Brain Architecture: From OpenClaw to Hermes

Jun 2026 · AI & Infrastructure

The surprising part of building a personal AI brain was not getting a model to answer questions. That part is easy now. The hard part was deciding what should remember, what should route, what should stay local, what should wake up at night, and what should leave me alone when I am using the machine for something else.

My first architecture was centered on OpenClaw. It gave the project the right question: what would a self-hosted personal assistant look like if it lived near my files, messages, and machines? The answer was useful, but it was not the final shape. Over time, the center moved from an assistant frontend to a routing and memory system. That is where Hermes became the live brain.

Quick answer: the local LLM setup that finally worked is not one giant model. It is Hermes as the self-hosted AI assistant orchestrator, a router that sends work to the right machine, local memory and governance files, privacy gates for sensitive context, and consumer hardware assigned to specific model roles.

The key shift: the assistant is not the compute layer. Hermes orchestrates, the router assigns work, and each machine has a narrow role.

The Problem Was Never One Model

Most local AI discussions collapse into a model debate. Which 7B? Which 12B? Which 35B? Which quant? I fell into that too. But once you actually run the system every day, the model is only one layer.

The better question is: what job is this model doing inside the whole system?

A fast always-on model is good for routine chat and scheduled work.
A larger reasoning model is better for hard synthesis, but not always better as the default.
A vision-capable model matters if screenshots and photos are part of the workflow.
A fallback model matters more than you think when the primary machine is busy, asleep, gaming, or restarting.

That changed the architecture from "find the smartest local model" to "give each machine a job and route intelligently."

Local LLM Setup: The Architecture That Stuck

The current version is simple at the top: Hermes owns the conversation surfaces, schedules, and orchestration. A router decides where each model call should go. The machines behind it specialize.

HermesOrchestrator

Owns the brain loopTelegram, web UI, scheduled jobs, source-of-truth notes, and tool behavior.

RouterTraffic controller

Chooses the right nodeDefault local route first, then overflow to idle machines, then fallback tiers only when needed.

LegionPrimary host

Main brain and router ownerAlways-on Windows GPU box. Handles the local default path and keeps the system coherent.

MaxClawHeavy reasoning

Fast overflow tierM1 Max node for larger text reasoning when it is awake, idle, and on power.

M5 AirVision and mobile

Image-aware specialistHandles vision and Apple Silicon inference without making the primary box carry every role.

ReefFallback

Always-on reserveSmaller GPU node for failover, batch work, and keeping the system responsive when the bigger nodes are unavailable.

The routing philosophy: local first, better idle machines second, reliable fallback third, cloud only as an explicit last resort for non-sensitive work.

Self-Hosted AI Assistant: What OpenClaw Taught Me

OpenClaw was valuable because it pushed the project out of "chat with a local model" and into "assistant as an operating surface." Skills, channels, message flows, file context, and scheduled work are the real product. A personal AI brain has to do more than answer a prompt in a browser tab.

But as the system matured, I needed a stronger separation between the assistant layer and the compute layer. The assistant should not be the whole architecture. It should be one consumer of a durable brain, memory, and routing substrate.

That is the move Hermes made possible. Hermes became the orchestrator. OpenClaw became part of the history and design vocabulary, not the live center of gravity.

OpenClaw vs Hermes: What Changed

Hermes made the architecture less magical and more operational. It owns the practical things that make the brain usable:

One command surface: Telegram, web UI, and scheduled jobs talk to the same brain instead of creating parallel assistants.
Explicit routing: calls can pin to a machine or use the automatic chain.
Local memory discipline: durable notes, changelogs, and governance files tell the system what changed and what is settled.
Privacy boundaries: personal and client-like content stays local; external verification is opt-in and topic-gated.
Machine empathy: laptops only take work when idle, on power, and healthy. Gaming or active use gets priority over background AI jobs.

That last point matters. A "personal AI brain" that ruins the rest of your computer is not intelligent infrastructure. It is just another background process with ambition.

Ollama, llama.cpp, and MLX Are Runtimes, Not the Architecture

The exact serving layer depends on the machine. A Windows GPU box may be better served by llama.cpp for long-context OpenAI-compatible routing. Apple Silicon often makes sense with MLX. Smaller fallback lanes can stay simple with Ollama. The point is not to crown one runtime forever. The point is to give the router a clean contract: this node is good at this job, is healthy right now, and is allowed to see this type of context.

Routing

Choose a machine by role, health, and load instead of letting every request queue behind the default model.

Memory

Promote decisions, preferences, and current state deliberately. Do not confuse raw logs with useful memory.

Privacy

Keep personal and sensitive work local by default. External checks should be explicit, topic-gated, and reversible.

Observability

Track which machine answered, whether it was healthy, and whether the answer was produced or merely timed out.

The Brain Is a System of Checks

The architecture that finally worked has more governance than glamour. There is a decision log for model changes. There is a context file that describes what is live now. There are router ledgers and fleet status checks. There are rules for not reopening settled model debates just because something new shipped.

That sounds boring until you have lived through the alternative: stale docs, mystery routes, silent fallbacks, models swapped for the wrong reason, and machines that look "reachable" but are not actually producing useful output.

The final architecture is not final because it will never change. It is final because the decision-making frame is stable:

Choose roles before choosing models.
Route around load instead of waiting behind it.
Keep sensitive context local by default.
Prefer observable boring systems over clever invisible ones.
Write down the decision so next week's experiment does not erase this week's lesson.

The Memory Layer Is the Next Frontier

The open question is not whether the brain can remember. It can. The question is what memory deserves to become durable.

A scratchpad is not memory. A transcript archive is not judgment. A vector store is not wisdom. The useful memory layer has to summarize, link, expire, and challenge itself. It needs to know the difference between a passing thought, a settled decision, a recurring preference, and a rule that protects privacy or quality.

That is where I am pushing next: less "store everything" and more "promote the right things." The same pattern applies. The winning design will not be one tool. It will be a workflow with boundaries.

My Takeaway

If you are building a personal AI brain, do not start by asking which model should be your brain. Start by designing the nervous system.

What receives the request? What context is allowed? What machine should answer? What happens if that machine is busy? What gets written down? What never leaves the house? What wakes up overnight? What is allowed to interrupt you?

Once those answers are clear, the model choices get easier. The architecture I settled on is Hermes as orchestrator, local routing as the nervous system, specialized consumer hardware as the compute layer, and written governance as the memory of why each choice exists.

A personal AI brain is not one big model. It is a set of decisions about attention, privacy, memory, and routing that happens to use models.

Related: Building a Multi-Machine Local AI Fleet and the free AI community where I share more of this build-out.