One of vSAN cluster experienced issue with SSD disk and once disk has been replaced capacity tier disk still belongs to old vSAN disk group.
To get this resolved first we need to indentify “old” SSD disk. This can be done using rvc
/127.0.0.1/Datacenter/computers/IDC/hosts> ls
0 esxp01 (host): cpu 2*8*2.09 GHz, memory 274.00 GB
1 esxp02 (host): cpu 2*8*2.09 GHz, memory 274.00 GB
2 esxp03 (host): cpu 2*8*2.09 GHz, memory 274.00 GB
3 esxp04 (host): cpu 2*8*2.09 GHz, memory 274.00 GB
/127.0.0.1/Datacenter/computers/IDC/hosts> vsan.disks_info 0
2021-04-17 05:53:46 +0000: Gathering disk information for host esxp01
2021-04-17 05:53:46 +0000: Done gathering disk information
Disks on host esxp01:
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| DisplayName | isSSD | Size | State |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local TOSHIBA Disk (naa.50000399c8482f41) | MD | 1117 GB | inUse |
| TOSHIBA AL15SEB120N | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local TOSHIBA Disk (naa.50000399c8482f89) | MD | 1117 GB | inUse |
| TOSHIBA AL15SEB120N | | | vSAN Format Version: vN/A |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local TOSHIBA Disk (naa.50000399c8482f0d) | MD | 1117 GB | inUse |
| TOSHIBA AL15SEB120N | | | vSAN Format Version: vN/A |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local TOSHIBA Disk (naa.50000399c8482f4d) | MD | 1117 GB | inUse |
| TOSHIBA AL15SEB120N | | | vSAN Format Version: vN/A |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local TOSHIBA Disk (naa.50000399c848313d) | MD | 1117 GB | inUse |
| TOSHIBA AL15SEB120N | | | vSAN Format Version: vN/A |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local TOSHIBA Disk (naa.50000399c8482ef9) | MD | 1117 GB | inUse |
| TOSHIBA AL15SEB120N | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local SEAGATE Disk (naa.5000c5009882d057) | MD | 1117 GB | inUse |
| SEAGATE ST1200MM0088 | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local TOSHIBA Disk (naa.50000399c8483109) | MD | 1117 GB | inUse |
| TOSHIBA AL15SEB120N | | | vSAN Format Version: vN/A |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local TOSHIBA Disk (naa.50000399c84831c5) | MD | 1117 GB | inUse |
| TOSHIBA AL15SEB120N | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local USB Direct-Access (eui.00a0504658335330) | MD | 29 GB | ineligible (Existing partitions found on disk 'eui.00a0504658335330'.) |
| Cypress RAID | | | |
| | | | Partition table: |
| | | | 5: 0.24 GB, type = vfat |
| | | | 6: 0.24 GB, type = vfat |
| | | | 7: 0.11 GB, type = coredump |
| | | | 8: 0.28 GB, type = vfat |
| | | | 9: 5.16 GB, type = coredump |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local SEAGATE Disk (naa.5000c5009883164f) | MD | 1117 GB | inUse |
| SEAGATE ST1200MM0088 | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local SEAGATE Disk (naa.5000c5009882b12f) | MD | 1117 GB | inUse |
| SEAGATE ST1200MM0088 | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local ATA Disk (naa.55cd2e404c25b806) | SSD | 447 GB | eligible |
| ATA INTEL SSDSC2BX48 | | | |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local TOSHIBA Disk (naa.50000399c84831dd) | MD | 1117 GB | inUse |
| TOSHIBA AL15SEB120N | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local SEAGATE Disk (naa.5000c5009882a7a3) | MD | 1117 GB | inUse |
| SEAGATE ST1200MM0088 | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local TOSHIBA Disk (naa.50000399c851c6f5) | MD | 1117 GB | inUse |
| TOSHIBA AL15SEB120N | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local ATA Disk (naa.55cd2e404c20424f) | SSD | 447 GB | inUse |
| ATA INTEL SSDSC2BB48 | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local TOSHIBA Disk (naa.5000039768098a85) | MD | 1117 GB | inUse |
| TOSHIBA AL14SEB120N | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Local ATA Disk (naa.55cd2e404c35970a) | SSD | 447 GB | inUse |
| ATA INTEL SSDSC2BX48 | | | vSAN Format Version: v7 |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
| Absent vSAN Disk (vSAN UUID:52a8ce51-bc37-253b-4246-80f0eede122d) | SSD | 0 GB | inUse |
| | | | vSAN Format Version: vN/A |
+-------------------------------------------------------------------+-------+---------+------------------------------------------------------------------------+
Next we can ssh to esxi host and check vsan disk 52a8ce51-bc37-253b-4246-80f0eede122d using vdq command and check disks that belongs to the same disk group
# list disk groups
(https://kb.vmware.com/s/article/2108910)
vdq -Hi
Mappings:
DiskMapping[0]:
SSD: 52a8ce51-bc37-253b-4246-80f0eede122d
MD: naa.50000399c8482f89
MD: naa.50000399c8482f0d
MD: naa.50000399c8482f4d
MD: naa.50000399c848313d
MD: naa.50000399c8483109
DiskMapping[2]:
SSD: naa.55cd2e404c20424f
MD: naa.50000399c8482f41
MD: naa.50000399c8482ef9
MD: naa.50000399c84831c5
MD: naa.50000399c84831dd
MD: naa.50000399c851c6f5
DiskMapping[4]:
SSD: naa.55cd2e404c35970a
MD: naa.5000c5009882d057
MD: naa.5000c5009883164f
MD: naa.5000c5009882b12f
MD: naa.5000c5009882a7a3
MD: naa.5000039768098a85
Once we know disk ID’s – we can check disk group UUID and if disk are being used by vSAN
MD: naa.50000399c8482f89
MD: naa.50000399c8482f0d
MD: naa.50000399c8482f4d
MD: naa.50000399c848313d
MD: naa.50000399c8483109
# check if disk is in use
esxcli vsan storage list -d naa.50000399c8482f89 | grep CMMDS
In CMMDS: false
esxcli vsan storage list -d naa.50000399c8482f0d | grep CMMDS
In CMMDS: false
esxcli vsan storage list -d naa.50000399c8482f4d | grep CMMDS
In CMMDS: false
esxcli vsan storage list -d naa.50000399c848313d | grep CMMDS
In CMMDS: false
esxcli vsan storage list -d naa.50000399c8483109 | grep CMMDS
In CMMDS: false
#check vSAN disk group UUID
esxcli vsan storage list -d naa.50000399c8482f89 | grep "Disk Group UUID"
VSAN Disk Group UUID: 52a8ce51-bc37-253b-4246-80f0eede122d
esxcli vsan storage list -d naa.50000399c8482f0d | grep "Disk Group UUID"
VSAN Disk Group UUID: 52a8ce51-bc37-253b-4246-80f0eede122d
esxcli vsan storage list -d naa.50000399c8482f4d | grep "Disk Group UUID"
VSAN Disk Group UUID: 52a8ce51-bc37-253b-4246-80f0eede122d
esxcli vsan storage list -d naa.50000399c848313d | grep "Disk Group UUID"
VSAN Disk Group UUID: 52a8ce51-bc37-253b-4246-80f0eede122d
esxcli vsan storage list -d naa.50000399c8483109 | grep "Disk Group UUID"
VSAN Disk Group UUID: 52a8ce51-bc37-253b-4246-80f0eede122d
Next we can check disk evacuation result and simply remove them once they are not being used
##1
# check evacuation result for disk in disk group 52a8ce51-bc37-253b-4246-80f0eede122d
[root@esxp01:~] esxcli vsan storage list -d naa.50000399c8482f89 | grep "VSAN UUID"
VSAN UUID: 5264fa92-5823-88a9-c86c-56fe18c94eb4
localcli vsan debug evacuation precheck -e 5264fa92-5823-88a9-c86c-56fe18c94eb4 -a noAction
Errors:
Not able to determine if 5264fa92-5823-88a9-c86c-56fe18c94eb4 is a disk/diskgroup/hostname uuid or its name.
# remove disk
esxcli vsan storage remove -u 5264fa92-5823-88a9-c86c-56fe18c94eb4
##2
esxcli vsan storage list -d naa.50000399c8482f0d | grep "VSAN UUID"
VSAN UUID: 523fc628-8dd2-a179-118a-946b8cefb8ff
localcli vsan debug evacuation precheck -e 523fc628-8dd2-a179-118a-946b8cefb8ff -a noAction
Errors:
Not able to determine if 523fc628-8dd2-a179-118a-946b8cefb8ff is a disk/diskgroup/hostname uuid or its name.
# remove disk
esxcli vsan storage remove -u 523fc628-8dd2-a179-118a-946b8cefb8ff
##3
esxcli vsan storage list -d naa.50000399c8482f4d | grep "VSAN UUID"
VSAN UUID: 52ea30ba-775c-59d1-ab6d-b70c11330d57
localcli vsan debug evacuation precheck -e 52ea30ba-775c-59d1-ab6d-b70c11330d57 -a noAction
Errors:
Not able to determine if 52ea30ba-775c-59d1-ab6d-b70c11330d57 is a disk/diskgroup/hostname uuid or its name.
# remove disk
esxcli vsan storage remove -u 52ea30ba-775c-59d1-ab6d-b70c11330d57
##4
esxcli vsan storage list -d naa.50000399c848313d | grep "VSAN UUID"
VSAN UUID: 5223cdd1-fb0d-93a7-02fa-83f78c5b459b
localcli vsan debug evacuation precheck -e 5223cdd1-fb0d-93a7-02fa-83f78c5b459b -a noAction
Errors:
Not able to determine if 5223cdd1-fb0d-93a7-02fa-83f78c5b459b is a disk/diskgroup/hostname uuid or its name.
# remove disk
esxcli vsan storage remove -u 5223cdd1-fb0d-93a7-02fa-83f78c5b459b
##5
esxcli vsan storage list -d naa.50000399c8483109 | grep "VSAN UUID"
VSAN UUID: 52a5def9-3518-d4e1-caf3-15c5efc4286d
localcli vsan debug evacuation precheck -e 52a5def9-3518-d4e1-caf3-15c5efc4286d -a noAction
Errors:
Not able to determine if 52a5def9-3518-d4e1-caf3-15c5efc4286d is a disk/diskgroup/hostname uuid or its name.
# remove disk
esxcli vsan storage remove -u 52a5def9-3518-d4e1-caf3-15c5efc4286d
Now it’s time to create new vSAN disk group
# check SSD that is not assigned to disk group (Is SSD: true )
esxcli storage core device list
...
SSD's
naa.55cd2e404c25b806
naa.55cd2e404c20424f
naa.55cd2e404c35970a
vdq -Hi
Mappings:
DiskMapping[0]:
SSD: naa.55cd2e404c20424f
MD: naa.50000399c8482f41
MD: naa.50000399c8482ef9
MD: naa.50000399c84831c5
MD: naa.50000399c84831dd
MD: naa.50000399c851c6f5
DiskMapping[2]:
SSD: naa.55cd2e404c35970a
MD: naa.5000c5009882d057
MD: naa.5000c5009883164f
MD: naa.5000c5009882b12f
MD: naa.5000c5009882a7a3
MD: naa.5000039768098a85
# Create new disk group
esxcli vsan storage add -s naa.55cd2e404c25b806 -d naa.50000399c8482f89 -d naa.50000399c8482f0d -d naa.50000399c8482f4d -d naa.50000399c848313d -d naa.50000399c8483109
# check that all is OK
vdq -Hi
Mappings:
DiskMapping[0]:
SSD: naa.55cd2e404c25b806
MD: naa.50000399c8482f89
MD: naa.50000399c8482f0d
MD: naa.50000399c8482f4d
MD: naa.50000399c848313d
MD: naa.50000399c8483109
DiskMapping[2]:
SSD: naa.55cd2e404c20424f
MD: naa.50000399c8482f41
MD: naa.50000399c8482ef9
MD: naa.50000399c84831c5
MD: naa.50000399c84831dd
MD: naa.50000399c851c6f5
DiskMapping[4]:
SSD: naa.55cd2e404c35970a
MD: naa.5000c5009882d057
MD: naa.5000c5009883164f
MD: naa.5000c5009882b12f
MD: naa.5000c5009882a7a3
MD: naa.5000039768098a85
VMware admin thoughts ... 