vmTools 10.3.x installation fails on server core/other os with failed to run CustomAction VM_PostInstall scripts

the VMinst.log/toolsinst.log did not have much information on the failure
Start by running msi debug logging:

msiexec /i "C:\MyPackage\toolsxxxx.msi" /L*V "C:\log\msi.log"
Logs: 
	MSI (s) (FC:F4) [08:04:42:754]: PROPERTY CHANGE: Adding VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19 property. Its value is '"C:\ProgramData\VMware\VMware CAF\pme\install\postInstall.bat" "C:\Program Files\VMware\VMware Tools\VMware CAF\pme\" "C
	:\ProgramData\VMware\VMware CAF\pme\"'.
	Action ended 8:04:42: VM_PostInstall_SD.A05FAB36_E570_4B23_8805_3633A16E8D19. Return value 1.
	MSI (s) (FC:F4) [08:04:42:754]: Skipping action: VM_StopVMwareProcs.869A7E00_8665_0000_83A8_EF0F76CF0001 (condition is false)
	MSI (s) (FC!8C) [08:04:46:207]: Closing MSIHANDLE (60) of type 790531 for thread 4492
	CustomAction VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19 returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
	MSI (s) (FC:BC) [08:04:46:207]: Closing MSIHANDLE (58) of type 790536 for thread 5876
	Action 8:04:46: Rollback. Rolling back action:
	Rollback: VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19
	MSI (s) (FC:F4) [08:04:46:238]: Executing op: ActionStart(Name=VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19,,)
	MSI (s) (FC:F4) [08:04:46:238]: Executing op: ProductInfo(ProductKey={F32C4E7B-2BF8-4788-8408-824C6896E1BB},ProductName=VMware Tools,PackageName={F32C4E7B-2BF8-4788-8408-824C6896E1BB}.msi,Language=1033,Version=167968773,Assignment=1,ObsoleteArg=0,ProductIcon=VmwareIcon,,PackageCode={66C1BF82-7ADE-472F-B0AE-1E6A85835452},,,InstanceType=0,LUASetting=0,RemoteURTInstalls=0,ProductDeploymentFlags=3)
	Rollback: Copying new files

in order to work this around, run the msi installer excluding vmware CAF

setup64.exe /S /v"/qn ADDLOCAL=ALL REMOVE=CAF /l*v C:\temp\vmtools-install.log" 

VMware tools install, command line

setup64.exe /S /v"/qn REBOOT=R ADDLOCAL=Audio,BootCamp,Hgfs,FileIntrospection,NetworkIntrospection,Perfmon,TrayIcon,Drivers,MemCtl,Mouse,MouseUsb,PVSCSI,EFIFW,SVGA,VMCI,VMXNet3,VSS,Toolbox,Plugins,Unity REMOVE=CAF,Audio,BootCamp,Hgfs,FileIntrospection,NetworkIntrospection,Perfmon,TrayIcon,Unity /l*v C:\temp\vmware1052_install.log" 

Fixing the broken/corrupt Locker Partition on Esxi

Start by determining the device backing up the locker partition

commands: 
ls -ltrh / | grep store
 vmkfstools -P /vmfs/volumes/5cdce747-375af1f6-b185-0050569674de
Output: 
[root@is-dhcp41-13:~] ls -ltrh / | grep store
lrwxrwxrwx    1 root     root           6 May 13 23:03 locker -> /store
lrwxrwxrwx    1 root     root          49 May 16 04:29 store -> /vmfs/volumes/5cdce747-375af1f6-b185-0050569674de


[root@is-dhcp41-13:~] vmkfstools -P /vmfs/volumes/5cdce747-375af1f6-b185-0050569674de
vfat-0.04 (Raw Major Version: 0) file system spanning 1 partitions.
File system label (if any):
Mode: private
Capacity 299712512 (36586 file blocks * 8192), 299712512 (36586 blocks) avail, max supported file size 0
Disk Block Size: 512/0/0
UUID: 5cdce747-375af1f6-b185-0050569674de
Partitions spanned (on "disks"):
        mpx.vmhba0:C0:T0:L0:8
Is Native Snapshot Capable: NO
[root@is-dhcp41-13:~]

Make a note of the device under the line Partitions spanned (on “disks”):

Note: The :8 on the above result signifives that this is partition 8 of the disk
Note: On a default install, the locker/tools iso are always stored to partition 8 of the installed disk/drive.

Format the partition with fat filesystem using the below command: Ensure you DO NOT MISS the partition number

vmkfstools -C vfat /dev/disks/mpx.vmhba0:C0:T0:L0:8
eg:
[root@is-dhcp41-13:~] vmkfstools -C vfat /dev/disks/mpx.vmhba0:C0:T0:L0:8
create fs deviceName:'/dev/disks/mpx.vmhba0:C0:T0:L0:8', fsShortName:'vfat', fsName:'(null)'
deviceFullPath:/dev/disks/mpx.vmhba0:C0:T0:L0:8 deviceFile:mpx.vmhba0:C0:T0:L0:8
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vfat file system on "mpx.vmhba0:C0:T0:L0:8" with blockSize 1048576 and volume label "none".
Successfully created new volume: 5cdcf45e-68f98eec-adb0-0050569674de


Note: If the format fails with the resource in use errors, the host will need a reboot.

re-create the symlink for store:

ln -snf /vmfs/volumes/5cdcf45e-68f98eec-adb0-0050569674de /store

ln -snf /vmfs/volumes/5cdcf45e-68f98eec-adb0-0050569674de /locker

Copy contents of the store partition from a working host, same Esxi build