Category: vCenter

screenshot:

Solution: remove/rename the software update status config file

mv /etc/applmgmt/appliance/software_update_state.conf /etc/applmgmt/appliance/software_update_state.conf.bak

On a side note: I ran into the issue after attempting to “stage and install” my homelabs vcsa,
logs reveal:

/var/log/vmware/applmgmt/applmgmt.log
2021-05-30T15:20:26.108 [3758]DEBUG:vmware.appliance.update.update_functions:Removing the mount point /mnt/iso-contents
2021-05-30T15:20:26.109 [3758]INFO:vmware.appliance.update.update_functions:ISO unmounted successfully
2021-05-30T15:20:26.109 [3758]DEBUG:vmware.appliance.update.update_b2b:discoverLocalUpdate failed.
Traceback (most recent call last):
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1434, in _discoverUpdateAt
    tempFolder)
  File "/usr/lib/python3.7/shutil.py", line 248, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/iso-contents/manifest-latest.xml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1640, in discoverLocalUpdatesNoException
    _discoverLocalUpdates()
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1631, in _discoverLocalUpdates
    _discoverUpdateAtIso()
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1607, in _discoverUpdateAtIso
    raise e
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1602, in _discoverUpdateAtIso
    _discoverUpdateAt(manifestDir, packagesDir, copyFileFunc, 'iso')
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1446, in _discoverUpdateAt
    raise RpmManifestNotFoundException
vmware.appliance.update.update_b2b.RpmManifestNotFoundException

so, I did the above, staged the upgrade repository this time:

Then ran the install:

Start TFTP service

service atftpd start

Allow TFTP port on the VCSA firewall

iptables -A port_filter -m state --state New -i eth0 -p udp --dport 69 -j ACCEPT

Confirm if the port is allowed on the firewall

iptables -nL | grep 69

Make the firewall rules persistent:

Export Ip tables rule

iptables-save > /etc/iptables.rules

Create a startup script at path: /etc/init.d/startftp.sh with the below contents:

#! /bin/sh
#
# TFTP Start/Stop the TFTP service and allow port 69
#
# chkconfig: 345 80 05
# description: atftpd

### BEGIN INIT INFO
# Provides: atftpd
# Required-Start: $local_fs $remote_fs $network
# Required-Stop:
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Description: TFTP
### END INIT INFO

service atftpd start
iptables-restore -c < /etc/iptables.rules

change the permissions of the script

chmod +x /etc/init.d/startftp.sh

set the script to run during startup:

chkconfig --add /etc/init.d/startftp.sh

copy the contents of TFTP from autodeploy_zip to /var/lib/tftpboot

If the vCenter was upgraded from 5.5, it retains legacy endpoints for lookup service
IE: https://FQDN:7444/lookupservce/sdk
Running certificate-manager will not replace the certificate and vCenter might actually start to complain about expired certificate although they are valid.

In this can be easily worked around by replacing the certificate on the VECS store:

Start by exporting the STS_INTERNAL_SSL_CERT and MACHINE_SSL_CERT

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /ssl/machine_ssl.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /ssl/machine_ssl.key


/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output /ssl/STS_INTERNAL_SSL_CERT-__MACHINE_CERT.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output /ssl/STS_INTERNAL_SSL_CERT-__MACHINE_CERT.key

Delete the contents of STS_INTERNAL_SSL_CERT store

 /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT -y

import machine_ssl store to the STS store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert  /ssl/machine_ssl.crt --key /ssl/machine_ssl.key 

Restart services and confirm service status

VMCA is the Default self-signed certificates that is set up at the time of vCenter deployment.

All solution users and machine SSL certificates are signed with this certificate.

VMCA certificates can be regenerated by using option 8 on the certificatae manager.

root@is-dhcp36-107 [ / ]# /usr/lib/vmware-vmca/bin/certificate-manager
                 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
                |                                                                     |
                |      *** Welcome to the vSphere 6.5 Certificate Manager  ***        |
                |                                                                     |
                |                   -- Select Operation --                            |
                |                                                                     |
                |      1. Replace Machine SSL certificate with Custom Certificate     |
                |                                                                     |
                |      2. Replace VMCA Root certificate with Custom Signing           |
                |         Certificate and replace all Certificates                    |
                |                                                                     |
                |      3. Replace Machine SSL certificate with VMCA Certificate       |
                |                                                                     |
                |      4. Regenerate a new VMCA Root Certificate and                  |
                |         replace all certificates                                    |
                |                                                                     |
                |      5. Replace Solution user certificates with                     |
                |         Custom Certificate                                          |
                |                                                                     |
                |      6. Replace Solution user certificates with VMCA certificates   |
                |                                                                     |
                |      7. Revert last performed operation by re-publishing old        |
                |         certificates                                                |
                |                                                                     |
                |      8. Reset all Certificates                                      |
                |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]:

choosing option 8, you are presented with the below options:

Option[1 to 8]: 8
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [[email protected]]:[email protected]
Enter password:

Please configure certool.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : US] :

Enter proper value for 'Name' [Default value : CA] :

Enter proper value for 'Organization' [Default value : VMware] :

Enter proper value for 'OrgUnit' [Default value : VMware Engineering] :

Enter proper value for 'State' [Default value : California] :

Enter proper value for 'Locality' [Default value : Palo Alto] :

Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] :

Enter proper value for 'Email' [Default value : [email protected]] :

Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : is-dhcp36-107.isl.vmware.com

Enter proper value for VMCA 'Name' :VMCA
Continue operation : Option[Y/N] ? : y

You are going to reset by regenerating Root Certificate and replace all certificates using VMCA
Continue operation : Option[Y/N] ? : y

Below are the .cfg’s created for the certificate

root@is-dhcp36-107 [ /var/tmp/vmware ]# ls -ltrh
total 32K
drwxr-xr-x 3 root root 4.0K Mar 14 00:07 cis-license
-rw-r--r-- 1 root root  191 Apr  2 19:24 certool.cfg
-rw-r--r-- 1 root root  243 Apr  2 19:24 vsphere-webclient.cfg
-rw-r--r-- 1 root root  240 Apr  2 19:24 vpxd-extension.cfg
-rw-r--r-- 1 root root  230 Apr  2 19:24 vpxd.cfg
-rw-r--r-- 1 root root   87 Apr  2 19:24 root.cfg
-rw-r--r-- 1 root root  217 Apr  2 19:24 MACHINE_SSL_CERT.cfg
-rw-r--r-- 1 root root  233 Apr  2 19:24 machine.cfg

Workflow (below are the commands that are run in the background)

/usr/lib/vmware-vmafd/bin/dir-cli service list --login [email protected] --password *****
1. machine-6368669f-591a-44fa-bfb3-a76b166bfed6
2. vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6
3. vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6
4. vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6

Create certificate cfg files for the respective services

• root.cfg

root@is-dhcp36-107 [ /var/tmp/vmware ]# cat root.cfg
Country = US
Name = VMCA
OrgUnit = VMware Engineering
State = California
#IPAddress =

• Solution users

root@is-dhcp36-107 [ /var/tmp/vmware ]# cat machine.cfg
Country = US
Name = machine-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = [email protected]
Hostname = is-dhcp36-107.isl.vmware.com


root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vsphere-webclient.cfg
Country = US
Name = vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = [email protected]
Hostname = is-dhcp36-107.isl.vmware.com


root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vpxd.cfg
Country = US
Name = vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = [email protected]
Hostname = is-dhcp36-107.isl.vmware.com



root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vpxd-extension.cfg
Country = US
Name = vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = [email protected]
Hostname = is-dhcp36-107.isl.vmware.com

• Machine SSL.cfg

root@is-dhcp36-107 [ /var/tmp/vmware ]# cat MACHINE_SSL_CERT.cfg
Country = US
Name = is-dhcp36-107.isl.vmware.com
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = [email protected]
Hostname = is-dhcp36-107.isl.vmware.com

Check if a backup store exist.. if not, create one.

/usr/lib/vmware-vmafd/bin/vecs-cli store list
MACHINE_SSL_CERT
TRUSTED_ROOTS
TRUSTED_ROOT_CRLS
machine
vsphere-webclient
vpxd
vpxd-extension
SMS

/usr/lib/vmware-vmafd/bin/vecs-cli store create --name BACKUP_STORE
service-control --start vmafdd
service-control --start vmcad
service-control --start vmdird

Export exiting certificate and import them to the backup store

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/vmware/old_machine_ssl.crt
/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp___MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp___MACHINE_CERT --cert /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.crt --key /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store machine --alias machine --output /storage/certmanager/rollback/machine_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /storage/certmanager/rollback/machine_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_machine --cert /storage/certmanager/rollback/machine_bkp.crt --key /storage/certmanager/rollback/machine_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vsphere-webclient --alias vsphere-webclient --output /storage/certmanager/rollback/vsphere-webclient_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /storage/certmanager/rollback/vsphere-webclient_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vsphere-webclient --cert /storage/certmanager/rollback/vsphere-webclient_bkp.crt --key /storage/certmanager/rollback/vsphere-webclient_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd --alias vpxd --output /storage/certmanager/rollback/vpxd_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /storage/certmanager/rollback/vpxd_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vpxd --cert /storage/certmanager/rollback/vpxd_bkp.crt --key /storage/certmanager/rollback/vpxd_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /storage/certmanager/rollback/vpxd-extension_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /storage/certmanager/rollback/vpxd-extension_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vpxd-extension --cert /storage/certmanager/rollback/vpxd-extension_bkp.crt --key /storage/certmanager/rollback/vpxd-extension_bkp.priv

Generate Machine_SSL certificate

/usr/lib/vmware-vmca/bin/certool --getrootca --server localhost
/usr/lib/vmware-vmca/bin/certool --selfca --config /var/tmp/vmware/root.cfg --server localhost
/usr/lib/vmware-vmca/bin/certool --getrootca --server localhost
/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/MACHINE_SSL_CERT.priv --pubkey=/storage/certmanager/MACHINE_SSL_CERT.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/MACHINE_SSL_CERT.priv --cert=/storage/certmanager/MACHINE_SSL_CERT.crt --config=/var/tmp/vmware/MACHINE_SSL_CERT.cfg
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store MACHINE_SSL_CERT --alias __MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store MACHINE_SSL_CERT --alias __MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert /storage/certmanager/MACHINE_SSL_CERT.crt --key /storage/certmanager/MACHINE_SSL_CERT.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store MACHINE_SSL_CERT --alias __MACHINE_CERT

Generate machine certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/machine.priv --pubkey=/storage/certmanager/machine.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/machine.priv --cert=/storage/certmanager/machine.crt --config=/var/tmp/vmware/machine.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/machine.crt --name machine-6368669f-591a-44fa-bfb3-a76b166bfed6 --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store machine --alias machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store machine --alias machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store machine --alias machine --cert /storage/certmanager/machine.crt --key /storage/certmanager/machine.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store machine --alias machine

Generate web client certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vsphere-webclient.priv --pubkey=/storage/certmanager/vsphere-webclient.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vsphere-webclient.priv --cert=/storage/certmanager/vsphere-webclient.crt --config=/var/tmp/vmware/vsphere-webclient.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vsphere-webclient.crt --name vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6 --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vsphere-webclient --alias vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vsphere-webclient --alias vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert /storage/certmanager/vsphere-webclient.crt --key /storage/certmanager/vsphere-webclient.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vsphere-webclient --alias vsphere-webclient

Generate vpxd certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vpxd.priv --pubkey=/storage/certmanager/vpxd.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vpxd.priv --cert=/storage/certmanager/vpxd.crt --config=/var/tmp/vmware/vpxd.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vpxd.crt --name vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6 --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd --alias vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vpxd --alias vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd --alias vpxd --cert /storage/certmanager/vpxd.crt --key /storage/certmanager/vpxd.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd --alias vpxd

Generate vpxd-extension certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vpxd-extension.priv --pubkey=/storage/certmanager/vpxd-extension.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vpxd-extension.priv --cert=/storage/certmanager/vpxd-extension.crt --config=/var/tmp/vmware/vpxd-extension.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vpxd-extension.crt --name vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6 --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd-extension --alias vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vpxd-extension --alias vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert /storage/certmanager/vpxd-extension.crt --key /storage/certmanager/vpxd-extension.priv

Update EAM and autodeploy with vpxd-extension certificate

/usr/bin/python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -s is-dhcp36-107.isl.vmware.com -c /storage/certmanager/vpxd-extension.crt -k /storage/certmanager/vpxd-extension.priv - [email protected] -p *****
/usr/bin/python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.rbd -s is-dhcp36-107.isl.vmware.com -c /storage/certmanager/vpxd-extension.crt -k /storage/certmanager/vpxd-extension.priv - [email protected] -p *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd-extension --alias vpxd-extension
service-control --stop --ignore  --all
service-control --start  --all

on Certain occations when the vCenter is upgraded, Plugins like pure, Netapp, NSX, SRM etc can cause incompatibility and prevent the webclient from loading up/Malfunctioning

Starting vSphere 6.7, we have a config file to white list and black list plugins which helps us to flag specific Malicious/incompatible plugins which prevents the web client form loading up

the symptoms to this include
* web client hangs at a blue, loading screen.
* web client loads but does not display any inventory object
* web client does not allow to click on the inventory objects (home button and task and events are sometimes clickable)

The quickest way to sort out these specious plugin is by adding the plugin id to blacklist to the respective UI/flex file comparability config.

UI(html5):

/etc/vmware/vsphere-ui/compatibility-matrix.xml

root@is-dhcp36-107 [ /etc/vmware/vsphere-ui ]# cat compatibility-matrix.xml
<!--
    This file lets you define a WHITE LIST and a BLACK LIST of plugins to control your own setup.
    It overrides the internal black and white lists that are hard-coded in this release.

    Fling Note: until further notice all plugins are disabled by the HTML5 client except SDK samples.
    Use this file to re-enable specific HTML plugins during your testing.
-->
<Matrix>
    <pluginsCompatibility>
        <!--
            WHITE LIST:
            Add this to enable all plugins whose plugin-package id is com.acme.example.myplugin:
                <PluginPackage id="com.acme.myplugin" status="compatible"/>
            Or this to specify all versions greater or equal to 2.1.0:
                <PluginPackage id="com.acme.myplugin" version=[2.1.0,] status="compatible"/>
            Or this to enable all plugins starting with com.acme:
                <PluginPackage id="com.acme.*" status="compatible"/>
        -->

        <!--
            BLACK LIST:
            Add this to disable a plugin whose plugin-package id is com.acme.example.myplugin:
                <PluginPackage id="com.acme.myplugin" status="incompatible"/>
         -->

    </pluginsCompatibility>
</Matrix>

vsphere-client (flex)

/etc/vmware/vsphere-client/compatibility-matrix.xml

root@is-dhcp36-107 [ /etc/vmware/vsphere-client ]# cat compatibility-matrix.xml
<!--
   The 'id' value allows for standard java regular expressions. The actual plugin id
   is matched against this regular expression.

   The 'version' values must be a string in any of the following formats. If skipped
   it indicates any version.

   Version: An exact version, e.g. 6.0.0, 5.5, etc. The format allows for maximum
      of 4 dotted numbers.

   Interval: (, 6.4], [6.5, 7.0), (7.5, ). Use empty strings to mark infinity values.

   Range: Several versions and intervals can be mixed in one single string
      producing a set of value, e.g. ( ,6.4), 6.8.3, 6.9.1, [7.5.2.2, ).

      Each item in the set is separated from the others using comma ",".
      Each item in the set can be either version or interval.

   The 'status' can be any of the following strings: unknown,
      incompatible, compatible, certified
-->
<Matrix>
  <pluginsCompatibility>
     <!--
        'Incompatible' plugins are not loaded. You can use this to 'blacklist'
        plugins if needed. To prevent specific plugin package(s) from
        loading, use the template entry below as guidance how to add new records
        with the actual id and version of the plugin you want to prevent from loading.

        The plugin ids can be taken from the plugin-package.xml file of each plugin.

        A few well known set of plugins locations (on the vCenter Appliance):
           /usr/lib/vmware-vsphere-client/plugin-packages/
           /etc/vmware/vsphere-client/vc-packages
           /etc/vmware/vsphere-client/cm-service-packages

        On Windows the following locations can be checked:
           <INSTALL DRIVE>\ProgramData\vCenterServer\runtime\vsphere-client\plugin-packages
           <INSTALL DRIVE>\ProgramData\vCenterServer\cfg\vsphere-client\vc-packages
           <INSTALL DRIVE>\ProgramData\vCenterServer\cfg\vsphere-client\cm-service-packages
     -->
     <!--
     <PluginPackage id="com.foo.plugin.id" version="1.0.0" status="incompatible"/>
     <PluginPackage id="net.bar.plugin.id" version="(,2.1]" status="incompatible"/>
     -->



     <!--
        The sample section below shows 'whitelist' definition. Compatible plugins
        are loaded. All others are declared as incompatible (due to the id regex),
        thus effectively forbidding them.

        The approach limits the list of plugins loaded only to a small 'white' list.
        This allows for vsphere-client to work in a 'safe-like' mode.

        Below are predefined sets of plugins for your convenience:
        1st-party, core:
          com.vmware.vsphere.client,
          com.vmware.ds,
          com.vmware.ssoadminui,
          com.vmware.vsphere.client.modules,
          com.vmware.license.client,
          com.vmware.opsmgmt.client

        1st-party extended (in addition to the above):
          com.vmware.loganalyzer,
          com.vmware.vsphere.client.telemetry

        2nd-party (basically anything that comes already pre-bundled with the
        vCenter Appliance and is not in the above two sets):
            com.vmware.vca.marketing.ngc.ui,
            com.vmware.vco
     -->
     <!--
     <PluginPackage id="com.vmware.vsphere.(client|ds|ssoadminui|client.modules)" status="compatible"/>
     <PluginPackage id="com.vmware.license.client" status="compatible"/>
     <PluginPackage id="com.vmware.opsmgmt.client" status="compatible"/>

     <PluginPackage id=".*" status="incompatible"/>
     -->
  </pluginsCompatibility>
</Matrix>

The plugin ID can be found on the extension manager under vCenter MOB. https://vc-fqdn/mob