One GPU, three containers, one owner.

Part 8 · local models, rebuilt

A week after part 6, the AI container worked, but it was doing three jobs at once: Ollama serving models, Open WebUI as the chat front end, and Jupyter for notebooks, all in one LXC container with the GPU. On 15 October 2026 it became three containers, Ollama was replaced, and the lab gained voice and images. The everyday model went from 60 to 80 tokens per second on the same card.

THE INSPIRATION

It started with a brainstorm note, written with another AI agent, about a new wave of inference engines for mixture-of-experts models: Strata, ik_llama.cpp, KTransformers and FreeToken. Their shared idea is to keep the experts a model uses most on the GPU, put the rest in system RAM, and lean on a fast SSD when even that runs out. The promise was bigger models and more tokens from the same 16 GB card.

The second push was simpler. One container held the chat UI, the model server and the notebooks, so restarting one meant touching all three, the disk filled with all three's images, and nothing said who the GPU belonged to at any moment.

THE GOAL

• Separate the jobs. One container per job, so each can be restarted, resized and backed up on its own.
• Give the GPU one owner. Every model, image and voice request goes through one place that decides what is loaded.
• Faster on the same hardware. More RAM was ruled out early: the spare sticks are 16 GB, and 32 GB DDR4 sticks cost too much.
• A wider lineup. Models from several families rather than mostly one, plus speech-to-text, text-to-speech, image generation and image editing in the same chat.

HOW IT WAS EVALUATED

The first pass was checking the brainstorm's claims against this machine before installing anything:

• An SSD does not add speed. Streaming weights from disk lets a bigger model load; it is the slowest tier, so it never raises tokens per second, and it does not lengthen the context either.
• The new engines only help a model that is partly on the CPU. The everyday model already fit entirely on the GPU, and two of the others were dense models, where expert placement does not apply. Only one model in the lineup stood to gain.
• The headline model did not fit. Strata is built for one 125-billion-parameter model, about 63 GB at 4-bit, on a host with 62 GB of RAM shared by every other guest. Parked until there is more RAM.
• Mainline llama.cpp had caught up. A week earlier it gained an expert cache that keeps hot experts on the GPU, the core trick of the specialist forks. So the first test was simply llama.cpp against Ollama.

The test: the same prompt of about 1,200 words, with a unique first line on every run so no cache could help, 400 tokens out, greedy decoding, the median of three runs, and only one engine allowed on the GPU at a time. Both engines got the same quantisation, flash attention and an 8-bit context cache.

The surprise was the first row: a model that was already 100% on the GPU ran a third faster under plain llama.cpp. The second surprise was practical. Ollama's model files did not load in upstream llama.cpp at all (one used a private architecture name, another a different tensor layout), so every model was downloaded again as the publisher's own GGUF file.

Then the lineup itself, chosen to span families, each tested the same way once the new containers existed:

The pattern from part 6 held. Mixture-of-experts models with a few billion active parameters stay quick even when they spill into RAM; dense models and anything bigger than RAM do not. Two other candidates never got as far as a download: Kimi K3, at 2.8 trillion parameters, and Aleph Alpha's German-first Kolibri-1, which stock llama.cpp cannot load yet.

THE ARCHITECTURE

llama-swap is the owner. It is a small proxy with one OpenAI-compatible address that starts the right llama-server for whichever model a request names, stops the previous one, and unloads after ten idle minutes. Two heavy requests can no longer race for 16 GB; they take turns. Speech runs beside it (Whisper for speech-to-text, Kokoro for text-to-speech), and ComfyUI handles images: FLUX.2 klein 4B to generate and Qwen-Image-Edit to edit.

Images and language models cannot both sit in 16 GB, so the two hand the card over explicitly. Before ComfyUI runs a job, it tells llama-swap to unload. Before llama-swap starts a model, it tells ComfyUI to free its memory. The workbench keeps direct GPU access because notebooks need it, and the same unload call clears the card before a training run.

The chat container needs no GPU at all. It is the only thing the browser talks to, and it reaches models, voice and images over the home network, so it can be rebuilt or upgraded without touching a single model.

HOW IT WAS BUILT

1. A new container that can see the GPU. Same recipe as part 6: the device lines from the old container's config, then the NVIDIA user-space libraries at the host's exact driver version, with no kernel module.

ON THE HOST

pct create 112 local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst \
  --hostname inference --cores 12 --memory 40960 --rootfs local-lvm:96 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.1.50/24,gw=192.168.1.1 \
  --unprivileged 1 --features nesting=1,keyctl=1 \
  --mp0 /mnt/oldroot/ollama/gguf,mp=/models,ro=1
# the GPU lines, copied from the old container
grep '^lxc\.' /etc/pve/lxc/107.conf >> /etc/pve/lxc/112.conf
pct start 112

INSIDE THE CONTAINER

# user space only, at the host's driver version
sh NVIDIA-Linux-x86_64-595.80.run --silent --no-kernel-module
# docker + the NVIDIA container toolkit, then let it run unprivileged
nvidia-ctk runtime configure --runtime=docker
sed -i 's/^#\?no-cgroups = .*/no-cgroups = true/' /etc/nvidia-container-runtime/config.toml
systemctl restart docker
docker run --rm --gpus all ubuntu nvidia-smi -L    # the card, by name

The disk started at 32 GB and filled during the first build: the CUDA images for these three services weigh about 20 GB together. It is 96 GB now, and models live on the NVMe drive, outside every backup.

2. llama-swap, with the hand-off built in. Each model is a command. The shared part asks ComfyUI to free the card first, then hands over to llama-server:

/OPT/LLAMA-SWAP/CONFIG.YAML (TWO OF FOUR MODELS)

macros:
  free_comfy: "curl -s -m 5 -X POST -H 'Content-Type: application/json' -d '{\"unload_models\":true,\"free_memory\":true}' http://192.168.1.50:8188/free >/dev/null;"
  srv: "exec /app/llama-server --port ${PORT} --jinja -fa on -ctk q8_0 -ctv q8_0 -t 6"
models:
  gpt-oss-20b:
    cmd: sh -c "${free_comfy} ${srv} -m /models/gpt-oss-20b-MXFP4.gguf -ngl 99 -c 32768"
    ttl: 600
  qwen3-coder-next:
    # 80B mixture of experts: experts in RAM, the hottest cached on the GPU
    cmd: sh -c "${free_comfy} ${srv} -m /models/Qwen3-Coder-Next-UD-IQ4_XS.gguf -c 32768 -ngl 99 --cpu-moe --moe-cache-mib 9000 --fit off"
    ttl: 600

For the 80B model, keeping every expert in RAM with a 9 GB cache of the busiest ones on the GPU decoded at 17.3 tokens per second, against 14.7 when llama.cpp placed layers on its own.

3. Voice. Speaches serves both directions behind an OpenAI-style API. On the CPU, Whisper took 22.5 seconds to transcribe a six-second clip on this ten-year-old Xeon. On the GPU it takes 0.68 seconds and about 1.3 GB, released after five idle minutes.

INSIDE THE CONTAINER

docker run -d --name speaches --restart unless-stopped --gpus all -p 8000:8000 \
  -e WHISPER__INFERENCE_DEVICE=cuda -e WHISPER__COMPUTE_TYPE=int8_float16 \
  -e STT_MODEL_TTL=300 -e TTS_MODEL_TTL=300 \
  ghcr.io/speaches-ai/speaches:latest-cuda

4. Images, and the other half of the hand-off. ComfyUI runs with --disable-smart-memory, so it moves models back to RAM after every job instead of holding the card. A ten-line custom node makes it ask llama-swap to unload before each job starts:

CUSTOM_NODES/GPU_GATE/__INIT__.PY

import urllib.request
import execution

_execute_async = execution.PromptExecutor.execute_async

async def execute_async(self, *args, **kwargs):
    try:
        urllib.request.urlopen("http://192.168.1.50:8080/unload", timeout=60).read()
    except Exception:
        pass    # never block an image on the gate
    return await _execute_async(self, *args, **kwargs)

execution.PromptExecutor.execute_async = execute_async
NODE_CLASS_MAPPINGS = {}

A 1024-pixel image from FLUX.2 klein takes 5 seconds warm, 14 from cold. An edit with Qwen-Image-Edit, quantised to 4-bit with a four-step speed-up, takes about 40 seconds, because a 13 GB model and its 9 GB text encoder come back onto the card for each job. That is the cost of always giving the GPU back, and worth it.

5. The chat container. Open WebUI's data moved across in one pipe, and the new container runs a pinned release (0.12.2, up from 0.11.3) rather than a moving tag:

ON THE HOST

pct exec 107 -- docker stop open-webui
pct exec 107 -- tar -C /opt/open-webui -cf - data | pct exec 113 -- tar -C /opt/open-webui -xf -

The lesson from part 6 applied again: Open WebUI keeps its settings in its own database, and those win over environment variables. The model connection, the voice endpoints and both image workflows were written as rows there, then the reverse proxy's entry for the chat name was pointed at the new container. No DNS changed.

6. Shrinking what was left. With Ollama and the old chat gone, the workbench went from 12 cores and 16 GB to 8 cores and 6 GB, and its disk from 90 GB used to 60. Three other containers came down to 2 GB each. Container memory limits used to add up to about 80 GB on a 62 GB host; now they roughly match it.

ON THE HOST

pct set 107 -cores 8 -memory 6144
pct fstrim 107     # hand freed blocks back to the thin pool

WHAT IS LEFT

The new ports still need status monitors, so the status page sees them. Strata stays parked until the host has more RAM, and the German model waits for upstream llama.cpp support. When either changes, adding a model is a download and three lines of config, which was the point.

ASK AN AGENT

PROMPT: SHOULD I REPLACE OLLAMA?

I run local models with Ollama on one GPU. Tell me, with measurements, whether llama.cpp behind llama-swap would be faster, and change nothing until I agree.

- List my models with their size, architecture (dense or mixture of experts) and how much of each sits on the GPU.
- Run llama-swap next to Ollama on another port. Download each model as the publisher's GGUF; do not assume Ollama's files will load.
- Benchmark both with the same prompt (unique first line each run), fixed output length, greedy decoding, median of three, and only one engine on the GPU at a time.
- For mixture-of-experts models that spill into RAM, compare automatic placement with --cpu-moe plus an expert cache.
Report a table of tokens per second for each model and engine, and which models are not worth keeping on my card.

THE SERIES

1. It started with a Raspberry Pi
2. One Ubuntu box and a macvlan network
3. Moving to Proxmox, one service per container
4. Four cores to sixteen
5. Sixteen days of backups that weren't
6. Choosing a 70-watt GPU
7. Twelve cores and most of a GPU
8. Building the lab with an agent in the loop
9. One GPU, three containers, one owner
10. From a browser kiosk to a 3 MB TUI