Twelve cores and most of a GPU.

Part 6 · local models

With a 16 GB card in the host (part 5) and sixteen cores under it (part 3), the last step was to let the AI container actually use them. On 8 October 2026 a day of tuning took a 35-billion-parameter model from 15 to about 36 tokens per second and made a 20B model the everyday default at 60. Here is every change, in the order it was made, with what each one bought.

A GPU INSIDE AN LXC CONTAINER

The host owns the card and runs the driver. An unprivileged container borrows it by being allowed to open the host's NVIDIA device files, which are bind-mounted in. Two kinds of line in the container's config do it: permission to use the devices, and the device files themselves.

ON THE HOST: FIND THE DEVICE NUMBERS

ls -l /dev/nvidia*
# typical output; note the two major numbers
# crw-rw-rw- 1 root root 195,   0 ... /dev/nvidia0
# crw-rw-rw- 1 root root 195, 255 ... /dev/nvidiactl
# crw-rw-rw- 1 root root 510,   0 ... /dev/nvidia-uvm      <- this major varies

/ETC/PVE/LXC/107.CONF (ADDED LINES)

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 510:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file

Inside, the container needs the NVIDIA user-space libraries at exactly the host's driver version, without a kernel module of its own, since it shares the host's kernel. Docker in the container then reaches the card through the NVIDIA container toolkit. If nvidia-smi works inside the container, everything after this works.

One gotcha worth knowing: if the container starts before the host has created the device files, the bind mounts create empty placeholder files instead, and the GPU silently is not there. pct reboot 107 after the host is fully up fixes it.

OLLAMA, CONFIGURED FOR ONE CARD

INSIDE THE CONTAINER

docker run -d --name ollama --restart unless-stopped \
  --gpus all \
  -p 11434:11434 \
  -v /srv/ollama:/root/.ollama \
  -e OLLAMA_HOST=0.0.0.0:11434 \
  -e OLLAMA_KEEP_ALIVE=24h \
  -e OLLAMA_MAX_LOADED_MODELS=2 \
  -e OLLAMA_FLASH_ATTENTION=1 \
  -e OLLAMA_KV_CACHE_TYPE=q8_0 \
  -e OLLAMA_CONTEXT_LENGTH=16384 \
  ollama/ollama:latest

• KEEP_ALIVE=24h: keep a model in memory once loaded, instead of unloading it after five idle minutes.
• MAX_LOADED_MODELS=2: room for an everyday model and one other, not a queue of them fighting over 16 GB.
• FLASH_ATTENTION and a q8_0 KV cache: the context cache takes roughly half the memory at negligible quality cost, which is what lets longer contexts stay on the GPU.

FOUR CHANGES, MEASURED

1. Persistence mode on the host. Small models felt slow even though they fit easily. With persistence mode off, the driver tears the GPU's state down when nothing is using it, and the next load paid about 23 seconds to wake it. One service on the host fixes it:

ON THE HOST

systemctl enable --now nvidia-persistenced
nvidia-smi -q | grep -i 'persistence mode'     # Enabled

2. Threads that match the cores. The container had six cores, but Ollama started sixteen CPU threads anyway. For a model entirely on the GPU that does not matter. For a model that spills past VRAM, the part on the CPU slowed to 1–2 tokens per second as sixteen threads fought over six cores. The fix is per model, a small Modelfile variant; I name them -lab:

TWO -LAB VARIANTS

# a 24B model that just fits: force every layer onto the GPU
cat > Modelfile.mistral <<'EOF'
FROM mistral-small3.2:24b
PARAMETER num_gpu 99
EOF
ollama create mistral-small3.2:24b-lab -f Modelfile.mistral

# a 35B mixture-of-experts model that cannot fit: pin the CPU threads
cat > Modelfile.qwen <<'EOF'
FROM qwen3.6:35b-a3b
PARAMETER num_thread 5
EOF
ollama create qwen3.6:35b-a3b-lab -f Modelfile.qwen

Counter-intuitively, more threads were slower: five was the sweet spot for the 35B model, even later with twelve cores to use.

3. Models off the backed-up disk. Model weights lived on the container's root disk, which was 82% full and copied into every nightly backup (the biggest contributor to part 4). They moved to a directory on the host's NVMe drive, bind-mounted in:

ON THE HOST: A BIND MOUNT FOR MODELS

mkdir -p /mnt/oldroot/ollama
# unprivileged container: its root is uid 100000 on the host
chown 100000:100000 /mnt/oldroot/ollama
pct set 107 -mp1 /mnt/oldroot/ollama,mp=/srv/ollama

Proxmox skips bind mounts when it backs a container up, so the container's setup is still backed up and the re-downloadable weights are not. The root disk went from 82% to 48%, and models load about three times faster from NVMe.

4. Twelve cores. With the new CPU there was room to double the container's share, live, with no restart:

ON THE HOST

pct set 107 -cores 12

That is the change that took the 35B model from 15–19 tokens per second to about 36. It runs 63% on the GPU; the rest is CPU, and the CPU now has the room.

WHAT 16 GB RUNS NOW

Every number below came from Ollama itself. --verbose prints the generation speed after each answer, and ollama ps shows how much of a loaded model is on the GPU:

INSIDE THE CONTAINER: MEASURING A MODEL

# run once to load it, then again for a warm number
docker exec -it ollama ollama run gpt-oss:20b-lab --verbose "Explain LVM-thin in three sentences."
# ...
# eval rate:            NN.NN tokens/s     <- the figure in the table

docker exec ollama ollama ps
# output looks like:
# NAME               SIZE     PROCESSOR          CONTEXT
# qwen3.6:35b-a3b-lab  ...    37%/63% CPU/GPU    16384

nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

Warm, one request at a time, after all four changes:

The pattern in that table is the lesson. A mixture-of-experts model only activates a few billion parameters per token, so it tolerates living partly on the CPU; a dense model of similar size that spills does not. On a 16 GB card, the biggest useful model is not the biggest that loads. Eight older models were deleted along the way.

Most of this was diagnosed in a conversation with an AI agent working on the lab directly. How that worked, and where it went wrong, is part 7.

ASK AN AGENT

PROMPT: MY LOCAL MODELS FEEL SLOW

My local models through Ollama feel slow. Find out why with measurements, and change nothing until I agree.

- The GPU: model, VRAM, driver, persistence mode, and what else is using it.
- The container or VM Ollama runs in: cores and memory it is given, and how many threads Ollama actually starts.
- Ollama's settings (keep-alive, flash attention, KV cache type, context length, loaded-model limit) and where model files are stored.
- For each model I use: size, how much is on the GPU (ollama ps), and warm tokens per second (ollama run --verbose), cold load time too.
Then propose changes one at a time, each with the number you expect it to move, and re-measure after each one.

THE SERIES

1. It started with a Raspberry Pi
2. One Ubuntu box and a macvlan network
3. Moving to Proxmox, one service per container
4. Four cores to sixteen
5. Sixteen days of backups that weren't
6. Choosing a 70-watt GPU
7. Twelve cores and most of a GPU
8. Building the lab with an agent in the loop
9. One GPU, three containers, one owner
10. From a browser kiosk to a 3 MB TUI