Sixteen days of backups that weren't.
Part 4 · Proxmox Backup Server
The point of moving to Proxmox was that every service could be snapshotted and backed up on its own. Proxmox Backup Server does that well: incremental, deduplicated backups of every guest, every night. Then for sixteen days it backed up nothing at all, and every dashboard in the lab stayed green. This post is how it is set up, how it failed, and what changed so it cannot fail quietly again.
THE SHAPE OF IT
proxmox host pbs container NAS
──────────── ───────────── ───
vzdump, 02:30 nightly ──► datastore ──NFS──► one share,
· every guest · dedup into chunks a quota on it
· snapshot mode · prune, daily
· garbage collection, weekly
- PBS runs as a container on the same Proxmox host. It is just another guest. That is fine for a home lab with the data on separate hardware; it would not be fine if the datastore lived on the same disks.
- The datastore is an NFS share on the NAS, exported to the host alone. The NAS enforces a quota on the share: first 100 GB, later raised to 400 GB.
- The host sees PBS as a storage target, so a backup job is a normal Proxmox job pointed at it.
SETTING IT UP
Attaching PBS to Proxmox is one storage entry; Proxmox needs the server, the datastore name, a user or API token, and the server's certificate fingerprint. Everything below in angle brackets is yours to fill in.
ON THE PROXMOX HOST: ADD PBS AS STORAGE
pvesm add pbs pbs-store \ --server <pbs-address> \ --datastore <datastore> \ --username '<user>@pbs!<token-name>' \ --password '<token-secret>' \ --fingerprint <sha256-fingerprint> pvesm status # the new storage should say "active"
The nightly job lives in /etc/pve/jobs.cfg and backs up every
guest in snapshot mode, so nothing is stopped to back it up:
/ETC/PVE/JOBS.CFG (THE RELEVANT LINES)
vzdump: backup-nightly
schedule 02:30
all 1
mode snapshot
storage pbs-store
prune-backups keep-daily=7,keep-weekly=4,keep-monthly=3
RUN ONE BY HAND
vzdump --all 1 --storage pbs-store --mode snapshot --mailnotification failure
# --mailnotification takes "always" or "failure"; "never" is rejected outright
SIXTEEN QUIET DAYS
From 22 September to 7 October 2026, every nightly backup failed. The
share's quota had filled. The first guest of each run died with
Disk quota exceeded while writing a chunk, and every guest
after it with syncfs: Input/output error.
None of that reached a screen. PBS's own task list showed only
connection error: not connected, which sounds like a network
blip, not a full disk. The monitor for PBS stayed green the whole time,
because it checked that the server answered, and the server did answer.
A backup server being up and backups working are different facts,
and only the first one was being measured.
WHERE THE REAL ERROR WAS: THE PROXMOX SIDE OF THE TASK
ls -t /var/log/pve/tasks/*/*vzdump* | head -3 grep -h -E 'ERROR|quota|Input/output' $(ls -t /var/log/pve/tasks/*/*vzdump* | head -1)
WHY IT FILLED
Two things together. Until the fix, the PBS side had no prune job and no garbage-collection schedule, so old chunks were never released. In PBS, pruning only forgets snapshots; space comes back only when garbage collection runs afterwards. And one guest was enormous. The AI container held about 101 GB, nearly all of it things that can be downloaded again: 56 GB of Docker images, 20 GB of build cache and 30 GB of model weights, all copied into the backup every night.
THE FIX
On the PBS side, a daily prune job and a weekly garbage collection. Both are easy to set in the web UI; this is the same thing on the command line, inside the PBS container:
IN THE PBS CONTAINER: PRUNE DAILY, COLLECT WEEKLY
proxmox-backup-manager prune-job create daily-prune \
--store <datastore> --schedule daily \
--keep-last 3 --keep-daily 7 --keep-weekly 4 --keep-monthly 3
proxmox-backup-manager datastore update <datastore> --gc-schedule 'sat 03:00'
# and once now, to get the space back today
proxmox-backup-manager garbage-collection start <datastore>
Garbage collection only frees chunks roughly a day after they were last referenced, so the space comes back the day after the prune, not immediately. Once it did, the datastore went from full to about 25% used. The model weights then moved out of the AI container onto a bind mount from the host, which Proxmox's backups skip by design; that is in part 6.
Two small scars remain from writing while full: two backup groups were
created with empty owner files, which PBS rejects with
parsing owner … not a valid user id until the file holds the
right token name or the empty group is removed.
MEASURING THE RIGHT THING
The lasting fix is that fullness is now a number someone sees. A root cron line on the host dumps storage usage every five minutes, and the unprivileged collector behind the public status page reads it and publishes how full the backup store is, red from 80%:
/ETC/CRON.D/LABPUSH-STORAGE ON THE HOST
*/5 * * * * root pvesh get /nodes/$(hostname)/storage --output-format json > /var/lib/labpush/storage.json.tmp && mv /var/lib/labpush/storage.json.tmp /var/lib/labpush/storage.json
Writing to a temporary file and renaming it means the collector never reads a half-written dump, and it ignores a dump older than thirty minutes, so a dead cron job shows up as missing data rather than as a frozen, healthy-looking number.
CHECKING IT YOURSELF
Three questions worth asking of any Proxmox backup setup, each one command:
ON THE PROXMOX HOST
# how full is the backup storage, really? pvesm status # is any guest left out of every backup job? pvesh get /cluster/backup-info/not-backed-up --output-format json # the newest backups on the store, with their times pvesm list pbs-store --content backup | sort -k2 | tail -20 # what did last night's job actually say? grep -h -E 'INFO: (Finished|Backup job finished)|ERROR' \ $(ls -t /var/log/pve/tasks/*/*vzdump* | head -1)
IN THE PBS CONTAINER
proxmox-backup-manager datastore list proxmox-backup-manager prune-job list proxmox-backup-manager garbage-collection status <datastore>
STILL OPEN
- No scheduled verify job yet. PBS can re-read every chunk on a schedule and check it against its checksum. It should, and it does not yet.
- No restore test yet. A backup that has never been restored is a hypothesis. The next step is to restore one guest to a spare ID and boot it.
BOTH, WHEN THEY ARE DONE
# in PBS: re-read every chunk weekly and check it against its checksum proxmox-backup-manager verify-job create weekly-verify \ --store <datastore> --schedule 'sun 04:00' --outdated-after 30 # on the host: restore a container's latest backup to a spare ID, unstarted pct restore 999 pbs-store:backup/ct/104/<timestamp> --storage local-lvm pct set 999 --delete net0 # same address as the original; start it with no network pct start 999 && pct exec 999 -- systemctl is-system-running pct destroy 999 --purge
The broader lesson goes beyond backups: a monitor that checks a service is up says nothing about whether the service is doing its job. The same week, the GPU was replaced. That is part 5.
ASK AN AGENT
PROMPT: ARE MY BACKUPS ACTUALLY WORKING?
Check whether my Proxmox backups are working, not whether the backup server is up. Read only. - For every guest: the time of its last successful backup, and the error from any failed run since. Read the Proxmox-side task logs, not just the PBS task list. - Datastore usage against its real limit, including any quota on the NFS or SMB share underneath it. - Whether prune, garbage collection and verify jobs exist, when they last ran, and what they freed. - Which guests are biggest in backups, and how much of that is re-downloadable (images, caches, model weights). - What would have alerted me if backups had stopped a week ago. If the answer is nothing, propose the smallest check that would.
THE SERIES
- It started with a Raspberry Pi
- One Ubuntu box and a macvlan network
- Moving to Proxmox, one service per container
- Four cores to sixteen
- Sixteen days of backups that weren't
- Choosing a 70-watt GPU
- Twelve cores and most of a GPU
- Building the lab with an agent in the loop
- One GPU, three containers, one owner
- From a browser kiosk to a 3 MB TUI