Deploying a vSAN Cluster

Accept the promiscuous mode.

Troubleshooting

vSAN data alarm 'vSAN object'

~] esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2026-10-10T12:44:44Z
Local Node UUID: 6ac9f1ee-fefe-7e75-9d9f-005056a9f8db
Local Node Type: NORMAL
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 6ac9f1ee-fefe-7e75-9d9f-005056a9f8db
Sub-Cluster Backup UUID: 6ac9f1df-0504-d098-6e06-005056a91f59
Sub-Cluster UUID: 52e66e3d-e38f-402f-4aa2-82e6f2c55107
Sub-Cluster Membership Entry Revision: 2
Sub-Cluster Member Count: 3
Sub-Cluster Member UUIDs: 6ac9f1ee-fefe-7e75-9d9f-005056a9f8db, 6ac9f1df-0504-d098-6e06-005056a91f59, 6ac9f1de-fe45-65dc-8f82-005056a9fb41
Sub-Cluster Member HostNames: ESXi01, ESXi03, ESXi02
Sub-Cluster Membership UUID: 7924ca6a-2005-533b-6a87-005056a9f8db
Unicast Mode Enabled: true
Maintenance Mode State: OFF
Config Generation: 53fc0dde-6fb5-48bd-a368-3069a9e0082e 5 2026-10-10T11:28:12.740
Mode: REGULAR

~] esxcli vsan health cluster list
Health Test Name Status
-------------------------------------------------- ------
Overall health red (vSAN Object health)
Data red
vSAN object health red
vSAN object format health green
Performance service red
Stats DB object green
Stats primary election red
Network green
Hosts with connectivity issues green
vSAN cluster partition green
All hosts have a vSAN vmknic configured green
vSAN: Basic (unicast) connectivity check green
vSAN: MTU check (ping with large packet size) green
vMotion: Basic (unicast) connectivity check green
vMotion: MTU check (ping with large packet size) green
Network latency check green
Physical disk green
Operation health green
Disk capacity green
Congestion green
Component limit health green
Component metadata health green
Memory pools (heaps) green
Memory pools (slabs) green
Cluster green
Advanced vSAN configuration in sync green
vSAN daemon liveness green
vSAN Disk Balance green
Resync operations throttling green
Software version compatibility green
Disk format version green
Capacity utilization green
Storage space green
Read cache reservations green
Component green
What if the most consumed host fails green

Solution:

Check vSAN network is ok on EXSi.

~] vmkping -I vmk1 -s 8972 -d 192.168.10.82
PING 192.168.10.82 (192.168.10.82): 8972 data bytes
8980 bytes from 192.168.10.82: icmp_seq=0 ttl=64 time=0.610 ms
8980 bytes from 192.168.10.82: icmp_seq=1 ttl=64 time=0.410 ms
8980 bytes from 192.168.10.82: icmp_seq=2 ttl=64 time=0.469 ms

--- 192.168.10.82 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.410/0.496/0.610 ms

Check inaccessible objects on VCSA.

~]# rvc administrator@vsphere.local:Welcome@123@localhost

> cd /localhost/Datacenter/computers/vSAN
/localhost/Datacenter/computers/vSAN> vsan.disks_stats .
2026-10-10 20:50:55 +0800: Fetching vSAN disk info from 192.168.3.82 (may take a moment) ...
2026-10-10 20:50:55 +0800: Fetching vSAN disk info from 192.168.3.81 (may take a moment) ...
2026-10-10 20:50:55 +0800: Fetching vSAN disk info from 192.168.3.83 (may take a moment) ...
2026-10-10 20:50:56 +0800: Done fetching vSAN disk infos
+---------------------+--------------+----------+------+-----------+--------+----------+----------+----------+----------+----------+---------+----------+----------+
| | | | Num | Capacity | | | Physical | Physical | Physical | Logical | Logical | Logical | Status |
| DisplayName | Host | DiskTier | Comp | Total | Used | Reserved | Capacity | Used | Reserved | Capacity | Used | Reserved | Health |
+---------------------+--------------+----------+------+-----------+--------+----------+----------+----------+----------+----------+---------+----------+----------+
| mpx.vmhba0:C0:T2:L0 | 192.168.3.81 | Cache | 0 | 100.00 GB | 0.00 % | 0.00 % | N/A | N/A | N/A | N/A | N/A | N/A | OK (v15) |
| mpx.vmhba0:C0:T1:L0 | 192.168.3.81 | Capacity | 5 | 99.99 GB | 2.59 % | 0.17 % | N/A | N/A | N/A | N/A | N/A | N/A | OK (v15) |
+---------------------+--------------+----------+------+-----------+--------+----------+----------+----------+----------+----------+---------+----------+----------+
| mpx.vmhba0:C0:T2:L0 | 192.168.3.82 | Cache | 0 | 100.00 GB | 0.00 % | 0.00 % | N/A | N/A | N/A | N/A | N/A | N/A | OK (v15) |
| mpx.vmhba0:C0:T1:L0 | 192.168.3.82 | Capacity | 5 | 99.99 GB | 2.78 % | 0.17 % | N/A | N/A | N/A | N/A | N/A | N/A | OK (v15) |
+---------------------+--------------+----------+------+-----------+--------+----------+----------+----------+----------+----------+---------+----------+----------+
| mpx.vmhba0:C0:T2:L0 | 192.168.3.83 | Cache | 0 | 100.00 GB | 0.00 % | 0.00 % | N/A | N/A | N/A | N/A | N/A | N/A | OK (v15) |
| mpx.vmhba0:C0:T1:L0 | 192.168.3.83 | Capacity | 5 | 99.99 GB | 1.87 % | 0.17 % | N/A | N/A | N/A | N/A | N/A | N/A | OK (v15) |
+---------------------+--------------+----------+------+-----------+--------+----------+----------+----------+----------+----------+---------+----------+----------+


/localhost/Datacenter/computers/vSAN> vsan.check_state .
2026-10-10 20:54:01 +0800: Step 1: Check for inaccessible vSAN objects
Detected 4 objects to be inaccessible
Detected 701bca6a-284e-0d0c-62b3-005056a9f8db on 192.168.3.81 to be inaccessible
Detected d705ca6a-0775-ddc8-dddf-005056a9f8db on 192.168.3.81 to be inaccessible
Detected 1821ca6a-6d36-6dd1-c2e0-005056a9fb41 on 192.168.3.81 to be inaccessible
Detected 631bca6a-6a8b-f4f0-0205-005056a9f8db on 192.168.3.81 to be inaccessible

2026-10-10 20:54:01 +0800: Step 2: Check for invalid/inaccessible VMs
Detected VM 'vCLS-c87d76aa-0ede-48f0-9a6e-c6d9c530f333' as being 'inaccessible'

2026-10-10 20:54:01 +0800: Step 3: Check for VMs for which VC/hostd/vmx are out of sync
Did not find VMs for which VC/hostd/vmx are out of sync


/localhost/Datacenter/computers/vSAN> vsan.object_info . 701bca6a-284e-0d0c-62b3-005056a9f8db
DOM Object: 701bca6a-284e-0d0c-62b3-005056a9f8db (v15, owner: 192.168.3.81, proxy owner: None, policy: No POLICY entry found in CMMDS)
RAID_1
Component: 701bca6a-7ee7-8c0c-3de8-005056a9f8db (state: ACTIVE (5), host: 192.168.3.81, capacity: mpx.vmhba0:C0:T1:L0, cache: mpx.vmhba0:C0:T2:L0,
votes: 1, usage: 0.3 GB, proxy component: false)
Component: 701bca6a-8d38-900c-c30f-005056a9f8db (state: ACTIVE (5), host: 192.168.3.82, capacity: mpx.vmhba0:C0:T1:L0, cache: mpx.vmhba0:C0:T2:L0,
votes: 1, usage: 0.3 GB, proxy component: false)
Witness: 701bca6a-34c2-920c-a4d6-005056a9f8db (state: ACTIVE (5), host: 192.168.3.83, capacity: mpx.vmhba0:C0:T1:L0, cache: mpx.vmhba0:C0:T2:L0,
votes: 1, usage: 0.0 GB, proxy component: false)
Extended attributes:
Address space: 2147483648B (2.00 GB)
Object class: vdisk
Object path: /vmfs/volumes/vsan:52e66e3de38f402f-4aa282e6f2c55107/631bca6a-6a8b-f4f0-0205-005056a9f8db/vCLS-c87d76aa-0ede-48f0-9a6e-c6d9c530f333.vmdk
Object capabilities: NONE

Set vCLS mode as "Retreat Mode" to delete vCLS VMs, and then return as "System Managed" to create new vCLS VMs.

vSAN -> Configure -> vSphere Cluster Services -> General -> EDIT VCLS MODE

vSAN performance service alarm 'Stats primary election'

The health check showed that 192.168.3.81 was the CMMDS Primary for this cluster, but no Stats Primary had been elected:

/localhost/Datacenter/computers/vSAN> vsan.perf.cluster_info .
+--------------+--------------+--------+
| Hostname | Role | Issues |
+--------------+--------------+--------+
| 192.168.3.82 | Agent | None |
| 192.168.3.81 | CMMDS Master | None |
| 192.168.3.83 | Agent | None |
+--------------+--------------+--------+


/localhost/Datacenter/computers/vSAN> vsan.check_state .
2026-10-10 22:25:40 +0800: Step 1: Check for inaccessible vSAN objects
Detected 5 objects to be inaccessible
Detected 701bca6a-284e-0d0c-62b3-005056a9f8db on 192.168.3.81 to be inaccessible
Detected 1821ca6a-6d36-6dd1-c2e0-005056a9fb41 on 192.168.3.81 to be inaccessible
Detected 631bca6a-6a8b-f4f0-0205-005056a9f8db on 192.168.3.81 to be inaccessible
Detected cc2cca6a-181c-59c4-6f43-005056a9fb41 on 192.168.3.83 to be inaccessible
Detected d705ca6a-0775-ddc8-dddf-005056a9f8db on 192.168.3.83 to be inaccessible

2026-10-10 22:25:40 +0800: Step 2: Check for invalid/inaccessible VMs

2026-10-10 22:25:40 +0800: Step 3: Check for VMs for which VC/hostd/vmx are out of sync
Did not find VMs for which VC/hostd/vmx are out of sync

/localhost/Datacenter/computers/vSAN> vsan.perf.stats_object_info .
Directory Name: .vsan.stats
vSAN Object UUID: 7443ca6a-fe80-03bf-9762-005056a9fb41
SPBM Profile: vSAN Default Storage Policy
vSAN Policy: hostFailuresToTolerate: 1, cacheReservation: 0, stripeWidth: 1, proportionalCapacity: 0, force Provisioning: 0, iopsLimit: 0, checksumDisabled: 0, subFailuresToTolerate: 0, spbmProfileGenerationNumber: 0, spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad, compressionDisabled: 0
vSAN Object Health: healthy

DOM Object: 7443ca6a-fe80-03bf-9762-005056a9fb41 (v15, owner: 192.168.3.82, proxy owner: None, policy: stri peWidth = 1, cacheReservation = 0, proportionalCapacity = [0, 100], hostFailuresToTolerate = 1, forceProvis ioning = 0, spbmProfileId = aa6d5a82-1c88-45da-85d3-3d74b91a5bad, spbmProfileGenerationNumber = 0, CSN = 1, spbmProfileName = vSAN Default Storage Policy)
RAID_1
Component: 7443ca6a-ed0e-b0bf-f5ac-005056a9fb41 (state: ACTIVE (5), host: 192.168.3.81, capacity: mpx.v mhba0:C0:T1:L0, cache: mpx.vmhba0:C0:T2:L0,
votes: 1, usage: 0.1 GB, proxy component: false)
Component: 7443ca6a-26cc-b3bf-8840-005056a9fb41 (state: ACTIVE (5), host: 192.168.3.83, capacity: mpx.v mhba0:C0:T1:L0, cache: mpx.vmhba0:C0:T2:L0,
votes: 1, usage: 0.1 GB, proxy component: false)
Witness: 7443ca6a-cb63-b6bf-e4d3-005056a9fb41 (state: ACTIVE (5), host: 192.168.3.82, capacity: mpx.vmhba 0:C0:T1:L0, cache: mpx.vmhba0:C0:T2:L0,
votes: 1, usage: 0.0 GB, proxy component: false)
Extended attributes:
Address space: 273804165120B (255.00 GB)
Object class: vmnamespace
Object path: /vmfs/volumes/vsan:52e66e3de38f402f-4aa282e6f2c55107/.vsan.stats
Object capabilities: NONE


/localhost/Datacenter/computers/vSAN> vsan.perf.stats_object_delete .
Deleting vSAN Stats DB object, which will stop vSAN Performance Service ...
Task: Disable vSAN performance service
New progress: 1%
Task result: error


#>> Reset advanced parameters on each host

[root@ESXi01:~] esxcfg-advcfg -g /vSAN/DOMPauseAllCCPs
Value of DOMPauseAllCCPs is 1
[root@ESXi01:~] esxcfg-advcfg -s 0 /vSAN/DOMPauseAllCCPs
Value of DOMPauseAllCCPs is 0

[root@ESXi01:~] esxcfg-advcfg -g /vSAN/IgnoreClusterMemberListUpdates
Value of IgnoreClusterMemberListUpdates is 1
[root@ESXi01:~] esxcfg-advcfg -s 0 /vSAN/IgnoreClusterMemberListUpdates
Value of IgnoreClusterMemberListUpdates is 0


#>> Disable perf service on vcsa rvc tool.

/localhost/Datacenter/computers/vSAN> vsan.perf.stats_object_delete .
Deleting vSAN Stats DB object, which will stop vSAN Performance Service ...
Task: Disable vSAN performance service
New progress: 1%
New progress: 50%
Task result: success


#>> Delete the .vsan.stats object on any host

~] esxcli vsan debug object list --all | grep -B1 "Directory Name: .vsan.stats"
Group UUID: d705ca6a-0775-ddc8-dddf-005056a9f8db
Directory Name: .vsan.stats
--
Group UUID: 1821ca6a-6d36-6dd1-c2e0-005056a9fb41
Directory Name: .vsan.stats

~] /usr/lib/vmware/osfs/bin/objtool delete -u d705ca6a-0775-ddc8-dddf-005056a9f8db -f
~] /usr/lib/vmware/osfs/bin/objtool delete -u 1821ca6a-6d36-6dd1-c2e0-005056a9fb41 -f


#>> Restart management services on all hosts

~] /etc/init.d/vsanmgmtd restart
Terminating watchdog process with PID 263537
vsanperfsvc stopped.
vsanperfsvc started.

Reference:

・ 'Stats Primary election' alert via vSAN Health Service for a vSAN Cluster
・ Deleting vSAN Stats DB Object in RVC Fails - VMware vSAN