Building a Multi-Machine Local AI Fleet

I run a fleet of machines that serve local LLMs 24/7 — no cloud API calls, no per-token billing, full control over the models and data. Here's the architecture and what I've learned building it.

The Fleet

Three machines, each with a role:

• Legion (Windows, RTX 4080) — the workhorse. Runs the primary inference server, handles the heaviest models, and acts as the orchestration controller for the fleet.
• Reef (Windows, 8GB GPU + 32GB RAM) — failover and parallel workloads. Runs quantized models via Ollama with partial GPU offload. Handles overflow when Legion is saturated.
• MacBook Air M5 — mobile inference node. Apple Silicon's unified memory makes it surprisingly capable for on-device LLM work when I'm away from the desk.

The Stack

The core inference layer is llama.cpp (via llama-server) and Ollama, depending on the machine and use case. On top of that:

• llama-router — a custom routing layer that directs requests to the right machine based on model availability, GPU utilization, and failover priority. If Legion is busy, Reef picks up the request automatically.
• Quantized models — Q8 for primary inference (quality matters), QAT 12B for failover (speed matters). The tradeoff between quality and throughput is real, and having both tiers means you don't have to choose.
• Gemma 4 12B — the current primary brain. Dense architecture, 100% GPU at ~42 tok/s on Legion. Handles vision tasks (OCR, image analysis) which rules out MoE alternatives that can't do multimodal.

What It Enables

The fleet runs a continuous processing loop — not batch jobs that run and exit, but always-on reactive loops that process topics when grounding files change or models upgrade. Think of it as a local AI that's always watching your project state and generating analysis.

Practical applications I run daily:

• Document analysis — feed in meeting transcripts, project docs, tracker exports. Get structured insights without sending anything to a cloud API.
• Multi-perspective review — run the same document through different model tiers or prompting strategies. Compare outputs. The fleet makes this cheap enough to do routinely.
• Grounded research loops — local models process against local context (project files, knowledge bases) in continuous cycles. Each pass deepens the analysis.

Lessons Learned

Concurrency kills

Running multiple inference requests against the same GPU without a lock will freeze the machine. I learned this the hard way — three concurrent requests hit the Q8 model simultaneously, mlock prevented OS memory reclaim, and the entire machine locked up. Fix: a shared lock primitive that serializes GPU access, removed mlock, capped context at 12K.

Quantization matters more than model size

A well-quantized 12B model at Q8 consistently outperforms a larger model at aggressive quantization. File size in the benchmark table tells you the quant level — always include it.

Failover is not optional

Machines go down. GPUs thermal throttle. WiFi drops. The router layer that automatically fails over to the next available machine is the difference between "my AI is down" and "my AI is slightly slower right now."

Local-only for sensitive data

Some analysis involves client data, personal context, or proprietary project state. Running inference locally means that data never leaves the network. This isn't paranoia — it's a real constraint when you're working with enterprise clients.

The value of a local fleet isn't cost savings over API calls — it's control. Control over what data touches which model, control over uptime, and control over the iteration speed when you're experimenting with new architectures.