cannot verify certificate chain. when attempting to add vCenter to usage meter 4.x

  • bump up logs to trace and re-adding shows the below:
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager55 | Sending package: sendDataId=435014645 idx1=0, length=187
[2020-09-24 09:59:31]  | TRACE |   Gnats MsgProcessor | tion.GNatsConnection.GNatsMessageHandler | Get a packet from gnats magicId=1974333149, sendDataId=942819630, idx1=0, totalLen=76
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | tion.GNatsConnection.GNatsMessageHandler | GNats 'gateway_cli' processing message '{"authorization":"auth_stab","data":[],"errCode":"OK","respond_id":"id_140"}' from 'gateway_cli.cl
ient.responds' with reply 'null'
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | ection.client.RequestManager.RequestInfo | Respond for trackingID - id_140  => errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: -------- Internal API call ------
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: command - vrni
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: action - read
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: request data - {"productType":"VRNI"}
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: trackingID - ProductManager56
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager56 | Gnats connection will send the data in 1 number of packages, id = 435014646, totalSize=186
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager56 | Sending package: sendDataId=435014646 idx1=0, length=186
[2020-09-24 09:59:31]  | TRACE |   Gnats MsgProcessor | tion.GNatsConnection.GNatsMessageHandler | Get a packet from gnats magicId=1974333149, sendDataId=942819631, idx1=0, totalLen=76
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | tion.GNatsConnection.GNatsMessageHandler | GNats 'gateway_cli' processing message '{"authorization":"auth_stab","data":[],"errCode":"OK","respond_id":"id_141"}' from 'gateway_cli.cl
ient.responds' with reply 'null'
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | ection.client.RequestManager.RequestInfo | Respond for trackingID - id_141  => errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 |    com.vmware.um.umconnection.UmResponse | ProductManager56 | Responding with: errCode=ERR_PM_WRONG_HOST_NAME errMsg=Certificate error for vcenter.abc.ee: Can not verify certificate chain errData=null JSonObj=null JSonArr=null

when looking at the certificate of the venter

openssl s_client -connect vcenter.abc.ee:443
Certificate chain
 0 s:/CN=vcenter.abc.ee
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
 2 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
...
...
...
-----END CERTIFICATE-----
subject=/CN=vcenter.abc.ee
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
---

Cause: Incorrect ordering of the certificates.

if you look closely on the certificate chain:
Certificate 0 is for the vcenter server and was issued by RapidSSL. Certificate 1 is the DigiCert root certificate. And certificate 2 is the RapidSSL certificate, issued by DigiCert.

Apparently, web servers are often forgiving of this kind of out-of-order certificate chain, but it does violate the SSL spec. Because certificate 0 is signed by RapidSSL, certificate 1 needs to be the RapidSSL certificate, which is currently certificate 2 instead.

To resolve this, re-import the custom certificate to vCenter server with the chain in the correct order.

ie:

 Certificate chain
 0 s:/CN=vcenter.abc.ee
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018 
  1  s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA  
 2  s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA 

Senario 2: You receive the error when attempting to add a product with custom certs or CA Signed certs.

vCenter:

For instance, if you are trying to add vCenter with CA/Custom root certificates, review the certificate chain imported on vCenter.

on the vCenter, run the below to review the certificate chain.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT

The certificate chain here must be in the correct order, Ie

-----BEGIN CERTIFICATE-----
signed certificate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
intermediate/subordinate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
intermediate/subordinate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
root
-----END CERTIFICATE-----

vCD

Review vCD certificates:

Keytool

Usage Meter 4.2 : trigger manual collection

manual collection can be triggered via API we first need to get a tioken and then invoke API:

token:

curl --location --request POST 'https://<UM_IP>:8443/api/v1/login' \
--header 'Content-Type: application/json' \
--data-raw '{
   "user":"usagemeter",
   "password":"vmware"
}' 

trigger collection:

curl --location --request POST 'https://<um_ip>:8443/api/v1/collect' \
--header 'Content-Type: application/json' \
--header 'sessionid: UmSIDzwOFhFRjBhdXOG6nrEaGJMco10t4im8pJN8kYXFn54E' \
--data-raw '{
  "trigger" : ["All"]
}'

Usage meter 4.2 ip/DNS changes back to default after reboot.

Symptom: On new deployment, the Ip/DNS are set correctly but after deployment and power on, the DNS or the IP range is different to that of what was used at the time of deployment

symptom2: When you try to change the Ip/DNS of the usage meter appliance, it returns back to the original values after reboot.

symptom 3: How to change usage meter 4.2 ip address/DNS

cause: usage meter 4.2 relies on ovfenv(a feature of vCenter that leverages IP pool’s to automatically lease out/Set DNS records on compatible ovf). The virtual machine port group used for usage meter is associated to a port group that has an IP pool associated with the incorrect values (values used here is what is cascaded over to usage meter)

Resolution: Log into vCenter flex client> networking>Look for the vm port group>configuration>network pool>associated network, edit this and correct the ip pool and the DNS there and then reboot/retry the deployment

Workaround: to disable (uncheck) ovf environment via vCenter flex client>edit UM VM settings>vapp options> under authoring>ip allocation>ip allocation scheme

Set the correct the IP/mask/gw/DNS in /opt/vmware/etc/vami/ovfEnv.xml and then reboot the VM

Note: This is only applicable if the VM port group at the time of deployment had an ip pool associated. If it did not have one, then you can set the ip/mask/gateway by following the instructions here: http://blog.ikigo.net/?p=472 or using vami_config_net

Replace Usage Meter 4.2 /4.3 certificates

Note: Before making any changes, it is highly recommended that you take the necessary backup/snapshots of the usage meter VM instances.

Usage meter certificate replacement is rather simple. Usage meter 4.2 Runs on Nginx and the Nginx configuration for the webserver is located here

/opt/vmware/cloudusagemetering/conf/nginx.conf

Let’s take a look at the Nginx configuration for certificate and the key used:

root@is-dhcp35-102 [ / ]# cat /opt/vmware/cloudusagemetering/conf/nginx.conf | grep crt
        ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
root@is-dhcp35-102 [ / ]# cat /opt/vmware/cloudusagemetering/conf/nginx.conf | grep key
        ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;

In order to replace the certificate, Either rename and replace the above-mentioned files with the new CA-signed certificate, key
OR
create a directory and dump the new, signed certificate, key in there and make necessary corrections to the config file.

Do keep in mind that the permissions will need to be set accordingly. (chown and chgrp)


root@is-dhcp35-102 [ /etc/ssl/certs ]# ls -ltrh nginx-selfsigned.crt
-rw-r--r-- 1 root root 1.3K Jun 25 05:02 nginx-selfsigned.crt

root@is-dhcp35-102 [ /etc/ssl/private ]# ls -ltrh
total 4.0K
-rw------- 1 usagemeter usagemeter 1.7K Jun 25 05:02 nginx-selfsigned.key

Once done, restart the usage meter appliance.

reboot -f

Usage meter 4.x root password reset/unlock

At the photon logo, press ‘e’ and you will see the editable grub menu:
Append rw init=/bin/bash on the line that starts with “linux”

Press ctrl + x or f10 to continue

you should see a screen similar to the below:

To unlock the account, type the below command (if you know the password)

/sbin/pam_tally2 -r -u root´╗┐

to reset the password, run the below

passwd root

Note: Changing the password does not unlock the account. if the account is locked out, you will need to run the previous command to unlock

Restart the guest of and then boot back into normal appliance and then try logging back in.