vRA Deployments Fail with IPAM Provider Error: {“code”:3000,”message”:”The extensible attributes for search are not specified. (Dynamic Script Module name : findFixedAddressesByEA#27)”} (Dynamic Script Module name : throwIpamError#28)

Posted on by Nithin Titta

After vRA was upgraded from 7.3 to 7.6 and the infoblox plugin to the latest, Deployments fail with the below message:

IPAM Provider Error: {"code":3000,"message":"The extensible attributes for search are not specified. (Dynamic Script Module name : findFixedAddressesByEA#27)"} (Dynamic Script Module name : throwIpamError#28)

After investigation from Infoblox, we found the Extensible attribute “VMware IPaddress” was missing on infoblox

Infoblox Documentation: https://docs.infoblox.com/download/attachments/8945695/Infoblox_IPAM_Plugin_for_VMware_vRA_User%27s_Guide.pdf?version=1&modificationDate=1606831097478&api=v2

In a second scenario, Looks like my DHCP range was fully allocated. we root caused this by reviewing the vRO workflow “allocate”

cannot verify certificate chain. when attempting to add vCenter to usage meter 4.x

Posted on by Nithin Titta
  • bump up logs to trace and re-adding shows the below:
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager55 | Sending package: sendDataId=435014645 idx1=0, length=187
[2020-09-24 09:59:31]  | TRACE |   Gnats MsgProcessor | tion.GNatsConnection.GNatsMessageHandler | Get a packet from gnats magicId=1974333149, sendDataId=942819630, idx1=0, totalLen=76
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | tion.GNatsConnection.GNatsMessageHandler | GNats 'gateway_cli' processing message '{"authorization":"auth_stab","data":[],"errCode":"OK","respond_id":"id_140"}' from 'gateway_cli.cl
ient.responds' with reply 'null'
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | ection.client.RequestManager.RequestInfo | Respond for trackingID - id_140  => errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: -------- Internal API call ------
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: command - vrni
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: action - read
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: request data - {"productType":"VRNI"}
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: trackingID - ProductManager56
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager56 | Gnats connection will send the data in 1 number of packages, id = 435014646, totalSize=186
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager56 | Sending package: sendDataId=435014646 idx1=0, length=186
[2020-09-24 09:59:31]  | TRACE |   Gnats MsgProcessor | tion.GNatsConnection.GNatsMessageHandler | Get a packet from gnats magicId=1974333149, sendDataId=942819631, idx1=0, totalLen=76
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | tion.GNatsConnection.GNatsMessageHandler | GNats 'gateway_cli' processing message '{"authorization":"auth_stab","data":[],"errCode":"OK","respond_id":"id_141"}' from 'gateway_cli.cl
ient.responds' with reply 'null'
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | ection.client.RequestManager.RequestInfo | Respond for trackingID - id_141  => errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 |    com.vmware.um.umconnection.UmResponse | ProductManager56 | Responding with: errCode=ERR_PM_WRONG_HOST_NAME errMsg=Certificate error for vcenter.abc.ee: Can not verify certificate chain errData=null JSonObj=null JSonArr=null

when looking at the certificate of the venter

openssl s_client -connect vcenter.abc.ee:443
Certificate chain
 0 s:/CN=vcenter.abc.ee
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
 2 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
...
...
...
-----END CERTIFICATE-----
subject=/CN=vcenter.abc.ee
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
---

Cause: Incorrect ordering of the certificates.

if you look closely on the certificate chain:
Certificate 0 is for the vcenter server and was issued by RapidSSL. Certificate 1 is the DigiCert root certificate. And certificate 2 is the RapidSSL certificate, issued by DigiCert.

Apparently, web servers are often forgiving of this kind of out-of-order certificate chain, but it does violate the SSL spec. Because certificate 0 is signed by RapidSSL, certificate 1 needs to be the RapidSSL certificate, which is currently certificate 2 instead.

To resolve this, re-import the custom certificate to vCenter server with the chain in the correct order.

ie: 

 Certificate chain
 0 s:/CN=vcenter.abc.ee
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018 
  1  s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA  
 2  s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA

Senario 2: You receive the error when attempting to add a product with custom certs or CA Signed certs.

vCenter:

For instance, if you are trying to add vCenter with CA/Custom root certificates, review the certificate chain imported on vCenter.

on the vCenter, run the below to review the certificate chain.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT

The certificate chain here must be in the correct order, Ie

-----BEGIN CERTIFICATE-----
signed certificate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
intermediate/subordinate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
intermediate/subordinate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
root
-----END CERTIFICATE-----

vCD

Review vCD certificates: 

Keytool

vRA 8.x GitLab Integration

Posted on by Nithin Titta

Create the GitLab integration in vRealize Automation:
1. Navigate to Infrastructure > Connections > Integrations.
2. Click + ADD INTEGRATION.
3. Select GitLab.
4. Enter the Name and Description.
5. Enter the server URL.
6. Enter the access token.
7. Click VALIDATE.
8. Click ADD.


The Access Token is copied from the GitLab console.

Preparing the GitLab Repository

You must create and save your blueprints in a specific structure in the GitLab repository so that vRealize Automation can detect them. The following prerequisites must be met:
• Create separate directories in the repository for each blueprint.
• Each directory must have one blueprint with the blueprint.yaml name.
• The blueprint must include the following properties at the top of the YAML code: 

— name:
— version:

Creating Dedicated Directories

In the GitLab console, navigate to Repository and click New directory. Specify the directory name and description.
To have 10 blueprints source controlled in a vRealize Automation project, you must create 10 separate directories.

Naming the Blueprints

You must meet the following requirements:
• Blueprint Name:
— Blueprint name must be blueprint.yaml.
— Name is case-sensitive.
— Only one blueprint exists in a directory.
• Blueprint Properties:
— First property must be name:.
— Second property must be version:.

If you fail to meet these requirements of blueprint name and properties in the GitLab repository, the blueprints are not detected in vRealize Automation.
Each repository creates a default branch called master. You can create additional branches. The branch name is used when you add a content source in vRealize Automation.

Creating the GitLab Content Source

Add a content source to the GitLab connection to import blueprints from the specified repository.

To add a content source to the GitLab integration
1. Select Infrastructure > Integrations and select the GitLab integration.
2. Select Projects.
3. Select an existing vRealize Automation project.
4. Select if you want to import Blueprints or ABX action scripts.
5. Enter the Repository path in GitLab. The path is the user name of the main GitLab account appended to the GitLab project or repository name.
6. Enter the GitLab branch to use.
7. (Optional) Enter a folder (directory) name. If the folder name is blank, all GitLab directories are available. When you click ADD, an automated synchronization task is initiated that imports blueprints into vRealize Automation. When the synchronization tasks are complete, a message indicates that the blueprints are imported.

vRealize Automation and GitLab Projects

The mapping between projects in GitLab and projects in vRealize Automation is one-to-one:
• For example, if you want to source control your blueprints for three projects in vRealize Automation, you must create three projects in GitLab.
• All the blueprints in vRealize Automation are tied with a vRealize Automation project.
• Selecting a vRealize Automation project is mandatory to create a GitLab content source.
• Selecting an existing vRealize Automation project (content source) with a new GitLab project results in a failure

Verifying the Blueprint Synchronization

You can verify the following blueprint synchronization:
• GitLab Integration: Project synchronization status
• GitLab Integration: Project synchronization history
• Cloud Assembly Design Tab: Imported blueprints

vRA 7.6 HP upgrade fails RPM already installed

Posted on by Nithin Titta

vRA 7.6 HP upgrade fails with 

2020-11-11T11:03:19.964428+01:00 srv01vraapp2t.corp.trumpf.com vcac-config: INFO  com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.isAllCommandExecuted:1085 - Checking if all commands a
re executed
2020-11-11T11:03:20.305569+01:00 srv01vraapp2t.corp.trumpf.com vcac-config: ERROR com.vmware.vcac.configuration.utils.ProcessUtil.execute:22 - Command /bin/sh returned invalid status 8. Output: Preparing...
                ##################################################
, Error:        package horizon-service-rpm-3.1.0.0-15448541.noarch (which is newer than horizon-service-rpm-3.1.0.0-15433743.noarch) is already installed
2020-11-11T11:03:20.305569+01:00 srv01vraapp2t.corp.trumpf.com vcac-config: ERROR com.vmware.vcac.cli.configurator.ConfiguratorImpl.processExceptionResult:160 - Command execution failed with unexpected erro
r: com.vmware.vcac.configuration.utils.ProcessUtil$ProcessExecutionException: package horizon-service-rpm-3.1.0.0-15448541.noarch (which is newer than horizon-service-rpm-3.1.0.0-15433743.noarch) is already
 installed.
com.vmware.vcac.configuration.utils.ProcessUtil$ProcessExecutionException: package horizon-service-rpm-3.1.0.0-15448541.noarch (which is newer than horizon-service-rpm-3.1.0.0-15433743.noarch) is already in
stalled
        at com.vmware.vcac.configuration.utils.ProcessUtil.execute(ProcessUtil.java:23) ~[vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
        at com.vmware.vcac.configuration.utils.ProcessUtil.executeShellCommand(ProcessUtil.java:96) ~[vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
        at com.vmware.vcac.cli.configurator.commands.cluster.patch.ExecuteShellCommand.execute(ExecuteShellCommand.java:28) ~[vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
        at com.vmware.vcac.cli.configurator.commands.cluster.ClusterPatchCommand.execute(ClusterPatchCommand.java:53) ~[vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
        at com.vmware.vcac.cli.configurator.ConfiguratorImpl.execute(ConfiguratorImpl.java:109) [vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
        at com.vmware.vcac.cli.configurator.Configurator.main(Configurator.java:123) [vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
2020-11-11T11:03:29.144546+01:00 srv01vraapp2t [database-failover-agent][3932]: 2020/11/11 11:03:29 --- AGENT next iteration ---
2020-11-11T11:03:29.145101+01:00 srv01vraapp2t [database-failover-agent][3932]: 2020/11/11 11:03:29 getAllVotes():, url suffix: api/master
2020-11-11T11:03:30.145235+01:00 srv01vraapp2t [database-failover-agent][3932]: 2020/11/11 11:03:30 ElectMaster(): Votes for Master:
Node localhost (On: true, Manual failover: false, IsLocalDbMaster: true) has 1 voters: localhost

Resolution

Run

  "rpm -Uvh --replacepkgs /usr/lib/vcac/patches/repo/cafe/patchRpms/*.rpm", --oldPackage

and then re-try the upgrade.

if the problem persists then 

del /usr/lib/vcac/patches/repo/cafe/patchRpms/horizon-service-rpm-3.1.0.0-15433743.noarch.rpm

and then re-run the upgrade

vRo 7/8 Plugin on vCenter 6.7 missing/do not load after upgrade

Posted on by Nithin Titta

There is no GA plugin available for the HTML5 client. This is planed to be included on the 8.2 release.

You may use the beta client as a workaround, However this needs manual installation.. Please ensure that you take a snapshot before you run through the below steps. 

cleanup: Delete/move the contents of the below directory. If the directory do not exist, create them.

ui client:
  /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vmware.vco-7.3.1
 Flex client:
  /etc/vmware/vsphere-client/vc-packages/vsphere-client-serenity/com.vmware.vco-7.3.1/

H5 plugin:

Download the zip file from 

https://my.vmware.com/group/vmware/downloads/get-download?downloadGroup=VCOIN-BETA

extract the contents of the file to the below path

/etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vmware.vco-7.3.1

set appropriate permissions to the directory

chown -R  vsphere-ui:users  /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vmware.vco-7.3.1

restart vsphere-ui client:

service-control --restart vsphere-ui

vsphere-flex-client

Download file:

https://communities.vmware.com/servlet/JiveServlet/download/35002-8-243022/vco-plugin-7.4.0.16380053.zip

extract the contents of the zip to

/etc/vmware/vsphere-client/vc-packages/vsphere-client-serenity/com.vmware.vco-7.3.1/

set appropriate permissions 

chown -R  vsphere-client:users /etc/vmware/vsphere-client/vc-packages/vsphere-client-serenity/com.vmware.vco-7.3.1

Note: for VRO 8.2, The path should be com.vmware.vco-7.4.0

restart vsphere-webclient service:

service-control --restart vsphere-client

log back in and double check , The plugin will take about 3-5 min to pull data from vRo on loading it.

Install Rancher and set up a Kubernetes cluster

Posted on by Nithin Titta

Install docker:

sudo curl https://releases.rancher.com/install-docker/19.03.sh | sh
sudo usermod -aG docker $user

install Rancher in a docker container (single node) with persistent data. We’ll map 80 of the container to 8080 of the docker host and 443 of the container to 8443 of the external host, I’m doing this since I will be using the above ports for different containers. 

docker run -d --restart=unless-stopped \
  -p 8080:80 -p 8443:443 \
  -v /opt/rancher:/var/lib/rancher \
  rancher/rancher:latest

check the status of the container:

docker ps -l

on a browser, open the IP address of the Docker host with the port specified previously. in my case it was https://192.168.3.101:8443/ , it should prompt you for a password similar to the below.

Once Done, You are prompted with the below page, click on “add a cluster” and then select “existing node”


Enter the cluster name and then hit next


In the node option, ensure that you select etcd, control plane and worker, copy the script from the below tab and run them on the Nodes that you want to spin up kubernetics.

In my senario since this is a lab setup, I will use the same docker host and 2 other nodes.

Node 1
Node 2
Node 3

Once Setup, wait a while for the nodes to startup (you will see few warnings, do not panic. wait it out.)

once they are all up, it should look like the below:

Congratulations!!, you have now set up rancher and are ready to start deploying Kubernetes workloads!!

Usage Meter 4.2 : trigger manual collection

Posted on by Nithin Titta

manual collection can be triggered via API we first need to get a tioken and then invoke API:

token:

curl --location --request POST 'https://<UM_IP>:8443/api/v1/login' \
--header 'Content-Type: application/json' \
--data-raw '{
   "user":"usagemeter",
   "password":"vmware"
}'

trigger collection:

curl --location --request POST 'https://<um_ip>:8443/api/v1/collect' \
--header 'Content-Type: application/json' \
--header 'sessionid: UmSIDzwOFhFRjBhdXOG6nrEaGJMco10t4im8pJN8kYXFn54E' \
--data-raw '{
  "trigger" : ["All"]
}'

Usage meter 4.2 ip/DNS changes back to default after reboot.

Posted on by Nithin Titta

Symptom: On new deployment, the Ip/DNS are set correctly but after deployment and power on, the DNS or the IP range is different to that of what was used at the time of deployment

symptom2: When you try to change the Ip/DNS of the usage meter appliance, it returns back to the original values after reboot.

symptom 3: How to change usage meter 4.2 ip address/DNS

cause: usage meter 4.2 relies on ovfenv(a feature of vCenter that leverages IP pool’s to automatically lease out/Set DNS records on compatible ovf). The virtual machine port group used for usage meter is associated to a port group that has an IP pool associated with the incorrect values (values used here is what is cascaded over to usage meter)

Resolution: Log into vCenter flex client> networking>Look for the vm port group>configuration>network pool>associated network, edit this and correct the ip pool and the DNS there and then reboot/retry the deployment

Workaround: to disable (uncheck) ovf environment via vCenter flex client>edit UM VM settings>vapp options> under authoring>ip allocation>ip allocation scheme

Set the correct the IP/mask/gw/DNS in /opt/vmware/etc/vami/ovfEnv.xml and then reboot the VM

Note: This is only applicable if the VM port group at the time of deployment had an ip pool associated. If it did not have one, then you can set the ip/mask/gateway by following the instructions here: http://blog.ntitta.in/?p=472 or using vami_config_net