Usage meter 4.5, SRM collection fails

SRM collector logs are written in vccol_main.log

/opt/vmware/cloudusagemetering/var/logs/vccol_main.log

2021-12-22 17:30:24.159 ERROR --- [vCenter collector thread] c.v.um.vccollector.srm.SRMCollector      : Unknown Failure  HTTP transport error: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46) :: https://srm02.ntitta.lab:443/drserver/vcdr/extapi/sdk
com.sun.xml.ws.client.ClientTransportException: HTTP transport error: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
        at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:102)
        at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:193)
        at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:115)
        at com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:109)
        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:1106)
        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:1020)
        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:989)
        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:847)
        at com.sun.xml.ws.client.Stub.process(Stub.java:433)
        at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:161)
        at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
        at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:62)
        at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:131)
        at com.sun.proxy.$Proxy41.srmLoginLocale(Unknown Source)
        at com.vmware.um.vccollector.srm.api.SrmApiClient.openSrmPort(SrmApiClient.java:116)
        at com.vmware.um.vccollector.srm.SRMCollector.getSrmLicenses(SRMCollector.java:312)
        at com.vmware.um.vccollector.srm.SRMCollector.collect(SRMCollector.java:119)
        at com.vmware.um.vccollector.srm.SRMCollectionStage.collectUsage(SRMCollectionStage.java:45)
        at com.vmware.um.vccollector.VCCollector.collectStages(VCCollector.java:276)
        at com.vmware.um.vccollector.VCCollector.collect(VCCollector.java:202)
        at com.vmware.um.vccollector.VCCollector.collect(VCCollector.java:32)
        at com.vmware.um.collector.CollectionHelper.collectFromServer(CollectionHelper.java:1014)
        at com.vmware.um.collector.CollectionHelper.collectFromServersWithReporting(CollectionHelper.java:1170)
        at com.vmware.um.collector.CollectionHelper.collectWithReporting(CollectionHelper.java:990)
        at com.vmware.um.collector.CollectionHelper.lambda$start$9(CollectionHelper.java:1518)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
        at org.bouncycastle.jsse.provider.ProvSSLSocketDirect.checkServerTrusted(ProvSSLSocketDirect.java:135)
        at org.bouncycastle.jsse.provider.ProvTlsClient$1.notifyServerCertificate(ProvTlsClient.java:360)
        at org.bouncycastle.tls.TlsUtils.processServerCertificate(TlsUtils.java:4813)
        at org.bouncycastle.tls.TlsClientProtocol.handleServerCertificate(TlsClientProtocol.java:784)
        at org.bouncycastle.tls.TlsClientProtocol.handleHandshakeMessage(TlsClientProtocol.java:670)
        at org.bouncycastle.tls.TlsProtocol.processHandshakeQueue(TlsProtocol.java:695)
        at org.bouncycastle.tls.TlsProtocol.processRecord(TlsProtocol.java:584)
        at org.bouncycastle.tls.RecordStream.readRecord(RecordStream.java:245)
        at org.bouncycastle.tls.TlsProtocol.safeReadRecord(TlsProtocol.java:843)
        at org.bouncycastle.tls.TlsProtocol.blockForHandshake(TlsProtocol.java:417)
        at org.bouncycastle.tls.TlsClientProtocol.connect(TlsClientProtocol.java:88)
        at org.bouncycastle.jsse.provider.ProvSSLSocketDirect.startHandshake(ProvSSLSocketDirect.java:445)
        at org.bouncycastle.jsse.provider.ProvSSLSocketDirect.startHandshake(ProvSSLSocketDirect.java:426)
        at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(Unknown Source)
        at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:89)
        ... 30 common frames omitted
Caused by: java.security.cert.CertificateException: There is no pinned certificate for this server and port
        at com.vmware.um.common.certificates.PinnedCertificateTrustManager.checkServerTrusted(PinnedCertificateTrustManager.java:81)
        at org.bouncycastle.jsse.provider.ImportX509TrustManager_7.checkServerTrusted(ImportX509TrustManager_7.java:56)
        at org.bouncycastle.jsse.provider.ProvSSLSocketDirect.checkServerTrusted(ProvSSLSocketDirect.java:131)
        ... 48 common frames omitted

Workaround:

Re-register SRM to vCenter using an IP instead of the FQDN

Navigate to https://SRM:5480, log in as admin and re-register. Change the local host field to the IP address from the dropdown

usage meter : vCenter collection fails with Failed customer rule collection – Could not find instanceUuid of the VM with moref: vm-2044 from VC server with id: 8.Could not find instanceUuid

vcenter collection fails with error:” Failed customer rule collection – Could not find instanceUuid of the VM with moref: vm-xxxx from VC server with id: x.Could not find instanceUuid”

cause: usage meter receives an error when attempting to query the UUID of the VM moref seen on the error.

on the vCenter inventory, you will also see stale/orphaned/inaccessible machines

Image_2021-11-04_12-28-08.png

Resolution: Unregister/remove the invalid vm’s from the vCenter inventory and wait for the next collection cycle (collection occurs every hour)

NSLCD based auth on Linux machine/Usage meter 4.4 LDAP

Below is the sample configuration to get UM-Ad intigation via LDAP (replace the contents of /ec/nslcd.conf with what is appropriate in your environment.

root@photon-machine [ /etc ]# cat /etc/nslcd.conf


uid nslcd
gid ldap

uri ldap://ntitta.lab
base dc=ntitta,dc=lab
binddn CN=service,CN=Users,DC=ntitta,DC=lab
bindpw P@ssw0rd

pagesize 1000
filter passwd (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
map    passwd uid              sAMAccountName
map    passwd homeDirectory    unixHomeDirectory
map    passwd gecos            displayName
filter shadow (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
map    shadow uid              sAMAccountName
map    shadow shadowLastChange pwdLastSet
filter group  (objectClass=group)

cat /etc/nsswitch.conf


hosts: files resolve dns
networks: files

protocols: files
services: files
ethers: files
rpc: files


passwd: files ldap
group: files ldap
shadow: files ldap

cat /etc/pam.d/system-auth

# Begin /etc/pam.d/system-auth
auth    required    pam_env.so
auth    required    pam_tally2.so onerr=fail deny=3 unlock_time=900 root_unlock_time=900 file=/var/log/tallylog
auth    sufficient  pam_ldap.so
auth    required    pam_unix.so
auth    optional    pam_faildelay.so delay=4000000

cat /etc/pam.d/system-account

# Begin /etc/pam.d/system-account
account    sufficient  pam_ldap.so
account    required    pam_tally2.so file=/var/log/tallylog
account    required    pam_unix.so
# End /etc/pam.d/system-account

cat /etc/pam.d/system-password

# Begin /etc/pam.d/system-password
password    sufficient  pam_ldap.so try_first_pass
password    requisite   pam_cracklib.so     minlen=10 minclass=4 difok=4 maxsequence=0 retry=3 enforce_for_root
password    requisite   pam_pwhistory.so    retry=3 remember=5 enforce_for_root
password    required    pam_unix.so         sha512 shadow use_authtok
# End /etc/pam.d/system-password

cat /etc/pam.d/system-session

# Begin /etc/pam.d/system-session
session   required    pam_unix.so
session   required    pam_limits.so
session   optional    pam_motd.so
session   optional    pam_lastlog.so showfailed
session   optional    pam_systemd.so
session   optional    pam_ldap.so
# End /etc/pam.d/system-session

cat /etc/pam.d/vmware-um-pam

auth       sufficient /lib64/security/pam_ldap.so
auth       required   /lib64/security/pam_unix_auth.so
account    sufficient /lib64/security/pam_ldap.so
account    required   /lib64/security/pam_unix_acct.so

aside from this, NSLCD also needs some pre-requisites for the user that is used to log in, This is described here: https://www.server-world.info/en/note?os=Windows_Server_2019&p=active_directory&f=12
basically the below attributes must exist on ad for the user in question:

uid           sAMAccountName
uidNumber     objectSid:<yourValue>
gidNumber     primaryGroupID
homeDirectory "/home/$sAMAccountName"
gecos         displayName
loginShell    "/bin/bash"
gidNumber      primaryGroupID

Note:

  • uid must be uniq for a user and should not be associated with any existing user in UM appliance, see getent passwd for the full list of used uids
  • gid must exist in the UM appliance before configuring AD. Use it to control what kind of privileges you want to give to the user. In most case use the usgemeger gid 1002

Note: Making changes to /etc/pam.d generally needs a reboot to take effect. Once the above config are in place, reboot the appliance and try login in
Note: If you move the original config files and create new files, please ensure the permissions of the files are corrected.

If configured correctly and when the user is attempted to log in on the UI, you should see the same domain user mapped in getent passwd

root@photon-machine [ /etc ]# getent passwd


root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/dev/null:/bin/false
daemon:x:6:6:Daemon User:/dev/null:/bin/false
messagebus:x:18:18:D-Bus Message Daemon User:/var/run/dbus:/bin/false
systemd-bus-proxy:x:72:72:systemd Bus Proxy:/:/bin/false
systemd-journal-gateway:x:73:73:systemd Journal Gateway:/:/bin/false
systemd-journal-remote:x:74:74:systemd Journal Remote:/:/bin/false
systemd-journal-upload:x:75:75:systemd Journal Upload:/:/bin/false
systemd-network:x:76:76:systemd Network Management:/:/bin/false
systemd-resolve:x:77:77:systemd Resolver:/:/bin/false
systemd-timesync:x:78:78:systemd Time Synchronization:/:/bin/false
nobody:x:65534:65533:Unprivileged User:/dev/null:/bin/false
sshd:x:50:50:sshd PrivSep:/var/lib/sshd:/bin/false
named:x:999:999::/var/lib/bind:/bin/false
polkitd:x:27:1000:PolicyKit Daemon Owner:/etc/polkit-1:/bin/false
nslcd:x:998:998:nslcd ldap user:/:/usr/sbin/nologin
ntp:x:87:87:Network Time Protocol:/var/lib/ntp:/bin/false
usagemeter:x:1000:1002::/home/usagemeter:/bin/bash
umauditor:x:1001:1003::/home/umauditor:/bin/bash
test:*:5000:1002:test:/home/test:/bin/bash                         <---------this is the domain user

Usage Meter 4.x picks up vRops although vRops no longer exist on the environment

usage meter 4.x picks up vRops from vCenter mob. after adding the vCenter , UM does dynamic discovery.  if for any reason, vRops extension is left stale behind on vCenter, UM will continue to think the product exists. 

To resolve this,
log in to the vCenter mob and remove com.vmware.vrops and com.vmware.vcops 

Instructions to remove the extensions can be found here: https://kb.vmware.com/s/article/1025360 , Kindly ensure that you remove only com.vmware.vrops and com.vmware.vcops.

Once the extension is removed, remove and re-add the product: vCenter (all vCenter’s referenced by this vRops)  back on to usage meter. you should no longer see vRops. 

adding vRA 7.6 to usage meter 4.3 fails with invalid credentials

Logs:

[2021-03-23 15:02:49]  | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 15. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED

[2021-03-23 15:02:57] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:02:57] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 15. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:03:21] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:03:21] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 15. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:14:17] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:14:17] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 15. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:19:50] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:19:50] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 16. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:47:09] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:47:09] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 16. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED

Resolution: Disable http2 on the IIS and then re-add the product



Instructions:
* Start → regedit
* Navigate to the folder/path: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters
* Under the Parameters folder, right-click the white-space, add 2 new DWORD (32-bit) values:
 – EnableHttp2Tls
 – EnableHttp2Cleartext
* Ensure both new values have been set to 0(disabled) by right-clicking the value and clicking “Modify…”
* Restart the OS.



Misc: To test iis credentials, you can perform the below curl command (run it from a different linux instance as UM has enforced FIPS which disables MD5 for ntlm based authentication)

curl --ntlm --user 'Administrator':'VMware1!' https://iis.ntitta.lab/repository/Data/ManagementModelEntities.svc/ProvisioningGroups -k



Once the above is done, you are likely to run into a second issue, Logs:

[2021-03-26 15:22:01]  |  WARN | t_credentials_create | vmware.um.common.http.LoggingInterceptor | GET /repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed with Too many follow-up requests: 21 Too many follow-up requests: 21

[2021-03-26 15:22:01] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: Too many follow-up requests: 21
[2021-03-26 15:22:01] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 5. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: Too many follow-up requests: 21=>Too many follow-up requests: 21

To resolve the above, Edit the vRA credentials in usage meter and under the IIS credentials, write the user name in the format: user@domain.com and leave the domain field blank

This image has an empty alt attribute; its file name is image-1024x808.png

if you see the below:

[2021-03-29 11:29:03]  |  INFO | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | Start test credentials for vRA Cafe server '16'

[2021-03-29 11:29:03] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | Server '16' failed to authenticate API endpoint identity/api/tokens on server 16 returned HTTP status 400

this means that the vRA credentials are incorrect. Try setting the user name as : administrator and then save.

cannot verify certificate chain. when attempting to add vCenter to usage meter 4.x

  • bump up logs to trace and re-adding shows the below:
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager55 | Sending package: sendDataId=435014645 idx1=0, length=187
[2020-09-24 09:59:31]  | TRACE |   Gnats MsgProcessor | tion.GNatsConnection.GNatsMessageHandler | Get a packet from gnats magicId=1974333149, sendDataId=942819630, idx1=0, totalLen=76
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | tion.GNatsConnection.GNatsMessageHandler | GNats 'gateway_cli' processing message '{"authorization":"auth_stab","data":[],"errCode":"OK","respond_id":"id_140"}' from 'gateway_cli.cl
ient.responds' with reply 'null'
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | ection.client.RequestManager.RequestInfo | Respond for trackingID - id_140  => errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: -------- Internal API call ------
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: command - vrni
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: action - read
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: request data - {"productType":"VRNI"}
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: trackingID - ProductManager56
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager56 | Gnats connection will send the data in 1 number of packages, id = 435014646, totalSize=186
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager56 | Sending package: sendDataId=435014646 idx1=0, length=186
[2020-09-24 09:59:31]  | TRACE |   Gnats MsgProcessor | tion.GNatsConnection.GNatsMessageHandler | Get a packet from gnats magicId=1974333149, sendDataId=942819631, idx1=0, totalLen=76
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | tion.GNatsConnection.GNatsMessageHandler | GNats 'gateway_cli' processing message '{"authorization":"auth_stab","data":[],"errCode":"OK","respond_id":"id_141"}' from 'gateway_cli.cl
ient.responds' with reply 'null'
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | ection.client.RequestManager.RequestInfo | Respond for trackingID - id_141  => errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 |    com.vmware.um.umconnection.UmResponse | ProductManager56 | Responding with: errCode=ERR_PM_WRONG_HOST_NAME errMsg=Certificate error for vcenter.abc.ee: Can not verify certificate chain errData=null JSonObj=null JSonArr=null

when looking at the certificate of the venter

openssl s_client -connect vcenter.abc.ee:443
Certificate chain
 0 s:/CN=vcenter.abc.ee
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
 2 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
...
...
...
-----END CERTIFICATE-----
subject=/CN=vcenter.abc.ee
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
---

Cause: Incorrect ordering of the certificates.

if you look closely on the certificate chain:
Certificate 0 is for the vcenter server and was issued by RapidSSL. Certificate 1 is the DigiCert root certificate. And certificate 2 is the RapidSSL certificate, issued by DigiCert.

Apparently, web servers are often forgiving of this kind of out-of-order certificate chain, but it does violate the SSL spec. Because certificate 0 is signed by RapidSSL, certificate 1 needs to be the RapidSSL certificate, which is currently certificate 2 instead.

To resolve this, re-import the custom certificate to vCenter server with the chain in the correct order.

ie:

 Certificate chain
 0 s:/CN=vcenter.abc.ee
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018 
  1  s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA  
 2  s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA 

Senario 2: You receive the error when attempting to add a product with custom certs or CA Signed certs.

vCenter:

For instance, if you are trying to add vCenter with CA/Custom root certificates, review the certificate chain imported on vCenter.

on the vCenter, run the below to review the certificate chain.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT

The certificate chain here must be in the correct order, Ie

-----BEGIN CERTIFICATE-----
signed certificate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
intermediate/subordinate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
intermediate/subordinate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
root
-----END CERTIFICATE-----

vCD

Review vCD certificates:

Keytool

Usage Meter 4.2 : trigger manual collection

manual collection can be triggered via API we first need to get a tioken and then invoke API:

token:

curl --location --request POST 'https://<UM_IP>:8443/api/v1/login' \
--header 'Content-Type: application/json' \
--data-raw '{
   "user":"usagemeter",
   "password":"vmware"
}' 

trigger collection:

curl --location --request POST 'https://<um_ip>:8443/api/v1/collect' \
--header 'Content-Type: application/json' \
--header 'sessionid: UmSIDzwOFhFRjBhdXOG6nrEaGJMco10t4im8pJN8kYXFn54E' \
--data-raw '{
  "trigger" : ["All"]
}'

Usage meter 4.2 ip/DNS changes back to default after reboot.

Symptom: On new deployment, the Ip/DNS are set correctly but after deployment and power on, the DNS or the IP range is different to that of what was used at the time of deployment

symptom2: When you try to change the Ip/DNS of the usage meter appliance, it returns back to the original values after reboot.

symptom 3: How to change usage meter 4.2 ip address/DNS

cause: usage meter 4.2 relies on ovfenv(a feature of vCenter that leverages IP pool’s to automatically lease out/Set DNS records on compatible ovf). The virtual machine port group used for usage meter is associated to a port group that has an IP pool associated with the incorrect values (values used here is what is cascaded over to usage meter)

Resolution: Log into vCenter flex client> networking>Look for the vm port group>configuration>network pool>associated network, edit this and correct the ip pool and the DNS there and then reboot/retry the deployment

Workaround: to disable (uncheck) ovf environment via vCenter flex client>edit UM VM settings>vapp options> under authoring>ip allocation>ip allocation scheme

Set the correct the IP/mask/gw/DNS in /opt/vmware/etc/vami/ovfEnv.xml and then reboot the VM

Note: This is only applicable if the VM port group at the time of deployment had an ip pool associated. If it did not have one, then you can set the ip/mask/gateway by following the instructions here: http://blog.ntitta.in/?p=472 or using vami_config_net

Replace Usage Meter 4.2 /4.3 certificates

Note: Before making any changes, it is highly recommended that you take the necessary backup/snapshots of the usage meter VM instances.

Usage meter certificate replacement is rather simple. Usage meter 4.2 Runs on Nginx and the Nginx configuration for the webserver is located here

/opt/vmware/cloudusagemetering/conf/nginx.conf

Let’s take a look at the Nginx configuration for certificate and the key used:

root@is-dhcp35-102 [ / ]# cat /opt/vmware/cloudusagemetering/conf/nginx.conf | grep crt
        ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
root@is-dhcp35-102 [ / ]# cat /opt/vmware/cloudusagemetering/conf/nginx.conf | grep key
        ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;

In order to replace the certificate, Either rename and replace the above-mentioned files with the new CA-signed certificate, key
OR
create a directory and dump the new, signed certificate, key in there and make necessary corrections to the config file.

Do keep in mind that the permissions will need to be set accordingly. (chown and chgrp)


root@is-dhcp35-102 [ /etc/ssl/certs ]# ls -ltrh nginx-selfsigned.crt
-rw-r--r-- 1 root root 1.3K Jun 25 05:02 nginx-selfsigned.crt

root@is-dhcp35-102 [ /etc/ssl/private ]# ls -ltrh
total 4.0K
-rw------- 1 usagemeter usagemeter 1.7K Jun 25 05:02 nginx-selfsigned.key

Once done, restart the usage meter appliance.

reboot -f

Usage meter 4.x root password reset/unlock

At the photon logo, press ‘e’ and you will see the editable grub menu:
Append rw init=/bin/bash on the line that starts with “linux”

Press ctrl + x or f10 to continue

you should see a screen similar to the below:

Update: UM 4.3+ you must remount root partition to read-write, run the below command:

mount -o remount,rw /

To unlock the account, type the below command (if you know the password)

/sbin/pam_tally2 -r -u root

to reset the password, run the below

passwd root

Note: Changing the password does not unlock the account. if the account is locked out, you will need to run the previous command to unlock

Restart the guest of and then boot back into normal appliance and then try logging back in.