Upgrade Proxmox VE 7 to 8: the real-world gotchas
Tested on
Date: 15 November 2026 · From: Proxmox VE 7.4 (Debian 11) · To: Proxmox VE 8.4.21 (Debian 12), kernel 6.8.12 Hardware: Dell PowerEdge R710 and R815, three Supermicro servers, one UEFI desktop-class host with a GPU. Five were in a cluster, one standalone.
Proxmox's own upgrade guide is good. This page is what it doesn't tell you: the six things that actually broke or slowed us down, in the order you'll meet them.
Before you start
- Have a way to reach the console (monitor and keyboard, IPMI/iDRAC, or a KVM). If a reboot doesn't come back, you'll need it to pick the old kernel.
- Back up your guest configs. At minimum copy
/etc/pve/qemu-server/and/etc/pve/storage.cfgsomewhere off the host. - Plan downtime. Every guest on the host stops during the upgrade. On a busy host, shutting guests down takes longer than the upgrade itself (see step 4).
- Cluster? Upgrade one node at a time and keep quorum. Don't mix major versions for longer than you need to.
1. Run the checker
pveversion # e.g. pve-manager/7.4-xx
pve7to8 --full
You want FAILURES: 0. A warning about running guests is normal at this point.
2. Fix your 7.4 repositories first (gotcha #1 and #2)
Bring 7.4 fully up to date before changing anything. On our hosts apt update failed straight away:
E: The repository 'http://security.debian.org bullseye-security Release' does not have a Release file.
Debian 11's security updates have moved to the archive. Point them there:
sed -i -E 's|^deb http://security.debian.org(/debian-security)? bullseye-security|deb http://archive.debian.org/debian-security bullseye-security|' /etc/apt/sources.list
Third-party repositories cause the same error. We had the PostgreSQL repository (apt.postgresql.org) for bullseye. Comment out anything that isn't Debian or Proxmox for now; you'll switch it to bookworm in step 5.
Databases
If a third-party repo provides your database server, make sure the bookworm repo gives you the same major version (for example PostgreSQL 15 to 15). A major-version jump needs its own planned migration. You can hold it with apt-mark hold.
Then:
apt update && apt dist-upgrade -y
pve7to8 --full # again: FAILURES: 0
3. (Optional) Pin your network interface names
A new kernel can rename network interfaces, and a renamed interface means a host you can't reach. We pinned ours to their MAC addresses before upgrading. Nothing was renamed, but it's cheap insurance:
# /etc/systemd/network/10-pin-eno1.link
[Match]
MACAddress=aa:bb:cc:dd:ee:01
[Link]
Name=eno1
One file per interface, using your own MAC addresses and names.
4. Shut down the guests, faster (gotcha #3)
Proxmox stops guests one at a time, waiting up to 180 seconds for each. On our host with 58 VMs that would have taken hours. Shutting them all down at once caused a different problem: some failed with lock timeouts.
What worked: shut down in parallel, then retry the stragglers. Do virtual machines (qm) and containers (pct) the same way, and keep your storage VM out of the parallel batch.
STORAGE_VM=100 # the VM other guests depend on (TrueNAS and the like); leave empty if you have none
qm list | awk -v s="$STORAGE_VM" 'NR>1 && $3=="running" && $1!=s {print $1}' > /root/running-vms.txt
pct list | awk 'NR>1 && $2=="running" {print $1}' > /root/running-cts.txt
[ -n "$STORAGE_VM" ] && qm list | awk -v s="$STORAGE_VM" '$1==s && $3=="running"{print $1}' > /root/running-storage-vm.txt
for id in $(cat /root/running-vms.txt); do qm shutdown "$id" --timeout 300 & done
for id in $(cat /root/running-cts.txt); do pct shutdown "$id" --timeout 120 & done
wait
qm list | awk -v s="$STORAGE_VM" 'NR>1 && $3=="running" && $1!=s {print $1}' # still running? repeat the loops above for these
pct list | awk 'NR>1 && $2=="running" {print $1}'
for id in $(cat /root/running-storage-vm.txt 2>/dev/null); do qm shutdown "$id" --timeout 600; done # the storage VM goes last
The storage VM is shut down last so the guests that use it have already stopped, and you start it first after the reboot (step 8), before anything that depends on it. The files in /root are your list of what was running, so you know what to start again.
How we checked these commands
The pct loop, the storage-VM exclusion and the ceph.list and third-party repo edits in step 5 were run in a lab with stand-ins that print the same column layout as qm list and pct list. They showed the right lists, all guests started shutting down in the same second, and the storage VM went last. We did not run the pct loop against live containers during our own upgrade, which used only the qm loop, so try it on one test container first.
5. Switch to bookworm and upgrade
sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list
sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list.d/pve*.list
# Ceph repository, only if you have one (see the Proxmox notes on upgrading Ceph first):
[ -f /etc/apt/sources.list.d/ceph.list ] && sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list.d/ceph.list
# Debian 12 security is back on the normal mirror:
sed -i 's|archive.debian.org/debian-security bookworm-security|security.debian.org/debian-security bookworm-security|' /etc/apt/sources.list
apt update
Turn the third-party repositories back on. In step 2 you commented them out. Switch each one to bookworm and uncomment it. This example is the PostgreSQL repository file, so use your own file name:
sed -i -E 's/^#\s*(deb .*)/\1/; s/bullseye/bookworm/g' /etc/apt/sources.list.d/pgdg.list
apt update
Check that the vendor publishes a bookworm repository first, and see the Databases warning in step 2 about keeping the same major version.
Run the upgrade so a dropped SSH session can't kill it (gotcha #4). One of our SSH sessions dropped mid-upgrade. Run it as a systemd unit instead:
systemd-run --unit=pve8-upgrade bash -c 'DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold dist-upgrade > /root/pve8-upgrade.log 2>&1'
systemctl is-active pve8-upgrade # "active" while running; finished when it isn't
tail -f /root/pve8-upgrade.log
--force-confold keeps your own versions of config files you've changed. If you'd rather answer the prompts yourself, run apt dist-upgrade in tmux or screen instead.
6. UEFI hosts: install systemd-boot before you reboot (gotcha #5)
On our one UEFI host, proxmox-boot-tool refresh failed after the upgrade because bootctl was missing. Rebooting like that risks a host that won't boot. Fix:
apt install systemd-boot
proxmox-boot-tool refresh
proxmox-boot-tool status
Legacy BIOS hosts booting with GRUB didn't have this problem. Check which you have with [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS.
7. Reboot and check
reboot
# after it's back:
pveversion # pve-manager/8.4.x
uname -r # 6.8.x-pve
zpool status -x # "all pools are healthy" if you use ZFS
8. Start the guests, and check whether they actually started
If you have a storage VM, start it first and wait until it is serving before you start anything else:
qm start $STORAGE_VM # then check its shares or disks are available before the next command
If you stopped or disabled the guest start service during the upgrade, guests marked "start at boot" may not come back by themselves. That happened to us on one host. Start them all with:
pvesh create /nodes/$(hostname)/startall
Templates stay "stopped"; that's normal.
9. Expect new SSH host keys on some VMs (gotcha #6)
After the reboot, three of our VMs presented new SSH host keys, and SSH warned REMOTE HOST IDENTIFICATION HAS CHANGED. All three had a cloud-init drive, and cloud-init regenerated the keys at boot.
Don't just accept the new key. Check it from the host through the QEMU guest agent, which doesn't go over the network:
qm guest exec <vmid> -- ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
If it matches what SSH is now showing you, update your known_hosts. Then stop it happening again by adding this to the VM's cloud-init configuration:
ssh_deletekeys: false
What went wrong, in short
| Problem | Fix |
|---|---|
bullseye-security has no Release file |
Point it at archive.debian.org |
Third-party repo blocks apt update |
Disable it for 7.4, switch it to bookworm for 8 |
| Guest shutdown takes hours, or parallel shutdowns hit lock timeouts | Shut down in parallel, then retry the stragglers |
SSH drop during dist-upgrade |
Run it with systemd-run or in tmux |
bootctl missing on a UEFI host |
apt install systemd-boot + proxmox-boot-tool refresh |
| Guests didn't start after the reboot | pvesh create /nodes/<node>/startall |
| New SSH host keys on cloud-init VMs | Check via the guest agent, then set ssh_deletekeys: false |
Next: Proxmox 8 to 9
We're doing 8 to 9 separately (the pve8to9 checker, then bookworm to trixie). That gets its own page once it's tested.