vRA Deployments Fail with IPAM Provider Error: {“code”:3000,”message”:”The extensible attributes for search are not specified. (Dynamic Script Module name : findFixedAddressesByEA#27)”} (Dynamic Script Module name : throwIpamError#28)

Posted on by Nithin Titta

After vRA was upgraded from 7.3 to 7.6 and the infoblox plugin to the latest, Deployments fail with the below message:

IPAM Provider Error: {"code":3000,"message":"The extensible attributes for search are not specified. (Dynamic Script Module name : findFixedAddressesByEA#27)"} (Dynamic Script Module name : throwIpamError#28)

After investigation from Infoblox, we found the Extensible attribute “VMware IPaddress” was missing on infoblox

Infoblox Documentation: https://docs.infoblox.com/download/attachments/8945695/Infoblox_IPAM_Plugin_for_VMware_vRA_User%27s_Guide.pdf?version=1&modificationDate=1606831097478&api=v2

In a second scenario, Looks like my DHCP range was fully allocated. we root caused this by reviewing the vRO workflow “allocate”

cannot verify certificate chain. when attempting to add vCenter to usage meter 4.x

Posted on by Nithin Titta
  • bump up logs to trace and re-adding shows the below:
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager55 | Sending package: sendDataId=435014645 idx1=0, length=187
[2020-09-24 09:59:31]  | TRACE |   Gnats MsgProcessor | tion.GNatsConnection.GNatsMessageHandler | Get a packet from gnats magicId=1974333149, sendDataId=942819630, idx1=0, totalLen=76
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | tion.GNatsConnection.GNatsMessageHandler | GNats 'gateway_cli' processing message '{"authorization":"auth_stab","data":[],"errCode":"OK","respond_id":"id_140"}' from 'gateway_cli.cl
ient.responds' with reply 'null'
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | ection.client.RequestManager.RequestInfo | Respond for trackingID - id_140  => errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: -------- Internal API call ------
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: command - vrni
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: action - read
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: request data - {"productType":"VRNI"}
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: trackingID - ProductManager56
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager56 | Gnats connection will send the data in 1 number of packages, id = 435014646, totalSize=186
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager56 | Sending package: sendDataId=435014646 idx1=0, length=186
[2020-09-24 09:59:31]  | TRACE |   Gnats MsgProcessor | tion.GNatsConnection.GNatsMessageHandler | Get a packet from gnats magicId=1974333149, sendDataId=942819631, idx1=0, totalLen=76
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | tion.GNatsConnection.GNatsMessageHandler | GNats 'gateway_cli' processing message '{"authorization":"auth_stab","data":[],"errCode":"OK","respond_id":"id_141"}' from 'gateway_cli.cl
ient.responds' with reply 'null'
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | ection.client.RequestManager.RequestInfo | Respond for trackingID - id_141  => errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 |    com.vmware.um.umconnection.UmResponse | ProductManager56 | Responding with: errCode=ERR_PM_WRONG_HOST_NAME errMsg=Certificate error for vcenter.abc.ee: Can not verify certificate chain errData=null JSonObj=null JSonArr=null

when looking at the certificate of the venter

openssl s_client -connect vcenter.abc.ee:443
Certificate chain
 0 s:/CN=vcenter.abc.ee
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
 2 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
...
...
...
-----END CERTIFICATE-----
subject=/CN=vcenter.abc.ee
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
---

Cause: Incorrect ordering of the certificates.

if you look closely on the certificate chain:
Certificate 0 is for the vcenter server and was issued by RapidSSL. Certificate 1 is the DigiCert root certificate. And certificate 2 is the RapidSSL certificate, issued by DigiCert.

Apparently, web servers are often forgiving of this kind of out-of-order certificate chain, but it does violate the SSL spec. Because certificate 0 is signed by RapidSSL, certificate 1 needs to be the RapidSSL certificate, which is currently certificate 2 instead.

To resolve this, re-import the custom certificate to vCenter server with the chain in the correct order.

ie: 

 Certificate chain
 0 s:/CN=vcenter.abc.ee
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018 
  1  s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA  
 2  s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA

Senario 2: You receive the error when attempting to add a product with custom certs or CA Signed certs.

vCenter:

For instance, if you are trying to add vCenter with CA/Custom root certificates, review the certificate chain imported on vCenter.

on the vCenter, run the below to review the certificate chain.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT

The certificate chain here must be in the correct order, Ie

-----BEGIN CERTIFICATE-----
signed certificate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
intermediate/subordinate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
intermediate/subordinate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
root
-----END CERTIFICATE-----

vCD

Review vCD certificates: 

Keytool