Eliminating single points of failure is the top priority when operating mission-critical workloads. Proxmox VE features built-in HA (High Availability) that automatically fails over VMs and containers to other nodes when a node goes down, minimizing service downtime. This article provides a detailed walkthrough of building and operating a Proxmox HA cluster.
For high availability across Kubernetes workloads, Kubo On-Premise is the ideal solution. Combining Proxmox HA with fully managed K8s delivers consistent availability from infrastructure through to applications.
HA Cluster Prerequisites
Based on the Proxmox official HA documentation, the following prerequisites must be met.
Hardware Requirements
- Minimum 3 nodes: Required for the quorum (majority vote) mechanism. 2-node setups can use a Corosync QDevice as a tiebreaker
- Shared storage: Storage accessible from all nodes (Ceph, NFS, iSCSI, or FC)
- Redundant network: Dedicated network for Corosync communication (10 GbE recommended)
- OOBM (Out-of-Band Management): Remote management interfaces such as IPMI, Dell iDRAC, or HPE iLO
- Hardware Watchdog: Hardware timer such as iTCO_wdt (Intel)
| Component | Role | Recommendation |
|---|---|---|
| Corosync | Inter-node communication and state sync | Dedicated 10 GbE network |
| Shared Storage | VM/CT data sharing | Ceph (recommended) / NFS |
| Fencing | Forced isolation of failed nodes | IPMI + Watchdog |
| QDevice | Tiebreaker for 2-node setups | Corosync QDevice |
Building the Cluster
Creating the Proxmox Cluster
Create the cluster on the first node, then join additional nodes:
# Create cluster on Node 1
pvecm create my-ha-cluster
# Join Nodes 2 and 3 (run on each joining node)
pvecm add 192.168.1.10 # Node 1's IP
# Verify cluster status
pvecm status
Configuring the Corosync Network
Corosync is the heartbeat of the cluster, managing inter-node communication and state synchronization. A reliable, low-latency network is essential. Packet loss or high jitter can cause node evictions and cluster instability.
Creating a separate bonded link exclusively for Corosync is strongly recommended:
# Check Corosync configuration
cat /etc/pve/corosync.conf
# Network interface bonding example
# Add to /etc/network/interfaces
auto bond0
iface bond0 inet static
address 10.10.10.1/24
bond-slaves eno3 eno4
bond-mode 802.3ad
bond-miimon 100
Corosync link settings can be managed from the Web UI under Datacenter, then Cluster. In production, always configure at least 2 links (rings) to eliminate single points of failure.
Configuring Shared Storage
HA resources must reside on shared storage accessible from all nodes. Ceph is the most recommended option for hyper-converged deployments:
# Install Ceph (run on each node)
pveceph install
# Create monitors (run on each node)
pveceph mon create
# Add OSDs
pveceph osd create /dev/sdb
pveceph osd create /dev/sdc
# Create Ceph pool
pveceph pool create vm-pool
Kubo automates complex distributed storage configuration and provides Kubernetes-optimized persistent volumes out of the box.
Configuring Fencing
Fencing is the most critical safety mechanism in an HA cluster. Fencing prevents the "split-brain" scenario where a failed node continues writing to shared storage. If an unresponsive node is actually still running and writing to shared disks while another node starts the same VM, data corruption will occur.
Watchdog Configuration
A hardware Watchdog automatically reboots a node when regular timer resets stop occurring:
# Configure Watchdog module
# /etc/default/pve-ha-manager
WATCHDOG_MODULE=iTCO_wdt
# Verify Watchdog operation
wdctl
# Falls back to softdog when hardware watchdog unavailable
modprobe softdog
IPMI Fencing
IPMI-based fencing can forcefully power off nodes over the network:
# Check node power status via IPMI
ipmitool -I lanplus -H 192.168.1.200 -U admin -P password chassis power status
# Force power off
ipmitool -I lanplus -H 192.168.1.200 -U admin -P password chassis power off
Critical: Out-of-band management (OOBM) is not a luxury in a Proxmox HA cluster -- it is a hard requirement for reliable fencing.
Managing HA Resources
Enabling HA
Place VMs and containers under HA management:
# Register VM 100 for HA
ha-manager add vm:100 --state started --max-restart 2 --max-relocate 2
# List HA resources
ha-manager status
# Modify HA resource settings
ha-manager set vm:100 --group prefer-node1
Node Affinity Rules
Since Proxmox VE 9.0, node affinity rules are the recommended approach, replacing HA groups:
- Node affinity: Restrict resources to specific nodes with priority ordering
- Resource affinity: Keep resources on the same node (positive) or separate nodes (negative)
- Strict mode: Prevents stopped resources from starting if target nodes are unavailable
Failover Policies
Resources support configurable recovery strategies:
| Parameter | Default | Description |
|---|---|---|
max_restart | 1 | Restart attempts on the same node |
max_relocate | 1 | Relocation attempts to different nodes after restarts fail |
Maintenance and Monitoring
Maintenance Mode
Before performing node maintenance, safely migrate HA services:
# Enable maintenance mode
ha-manager crm-command node-maintenance enable pve-node1
# Wait for all HA services to migrate off the node
# Perform maintenance work after migration completes
# Disable maintenance mode
ha-manager crm-command node-maintenance disable pve-node1
Rolling Updates
Update clusters one node at a time -- never update all nodes simultaneously:
- Set the node to maintenance mode
- Confirm HA service migration completes
- Run
apt update && apt full-upgrade -y - Reboot and verify the node rejoins the cluster
- Proceed to the next node
Failover Testing
Always test failover before production deployment:
# Test with the simulator
apt install pve-ha-simulator
pve-ha-simulator
# Manual failover (migrate VM to another node)
ha-manager migrate vm:100 pve-node2
# Monitor HA status
ha-manager status
The Proxmox forum hosts active discussions about HA configuration best practices.
Conclusion
A properly designed Proxmox HA cluster delivers enterprise-grade availability. The keys are redundant Corosync networking, reliable fencing for safe node isolation, and trustworthy shared storage.
For integrated infrastructure management that includes Kubernetes workload HA, consider Kubo On-Premise. Deploy fully managed K8s on top of Proxmox HA to guarantee automatic Kubernetes workload recovery during node failures.
For consultation on HA cluster design and implementation, contact us for expert guidance.
Related Links: