Saltstack + vSphere: Deploying Windows VM’s with Windows Minion

Ensure that you have set up sphere provider provider, refer my previous blog https://blog.ntitta.in/?p=597

create a windows profile

/etc/salt/cloud.profiles.d/w16k.conf 

root@saltyub:/# cat /etc/salt/cloud.profiles.d/w16k.conf 
w16k:
  provider: vcsa
  clonefrom: w16k_salt 
#  devices: 
#   network: 
#    Network adaptor 1:
#     name: VM Network
#     adapter_type: vmxnet3
#     switch_type: standard
#     ip: 172.16.70.79
#     gateway: [172.16.1.1]
#     subnet_mask: 255.255.128.0
#     domain: ntitta.lab
  cluster: vSAN
  datastore: vsanDatastore
  power_on: True
  deploy: True
  customization: True
  minion:
   master: saltyu.ntitta.lab
  win_username: administrator 
  win_password: 'P@ssw0d'
  plain_text: True
  win_user_fullname: admin
  win_run_once: 'powershell.exe c:\scripts\e.winrm.ps1'
  win_installer: /salt/minion/Salt-Minion-3000.9-Py2-AMD64-Setup.exe
  winrm_verify_ssl: False

Ensure that you have the smbprotocol and pypsexec installed

pip3 install smbprotocol
pip3 install pypsexec

on the guest windows server template, ensure vmware tools is installed and create a PowerShell script in the path: c:\scripts\e.winrm.ps1, refer salt doc for more information: https://docs.saltproject.io/en/latest/topics/cloud/windows.html

New-NetFirewallRule -Name "SMB445" -DisplayName "SMB445" -Protocol TCP -LocalPort 445
New-NetFirewallRule -Name "WINRM5986" -DisplayName "WINRM5986" -Protocol TCP -LocalPort 5986

winrm quickconfig -q
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="300"}'
winrm set winrm/config '@{MaxTimeoutms="1800000"}'
winrm set winrm/config/service/auth '@{Basic="true"}'

$SourceStoreScope = 'LocalMachine'
$SourceStorename = 'Remote Desktop'

$SourceStore = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Store -ArgumentList $SourceStorename, $SourceStoreScope
$SourceStore.Open([System.Security.Cryptography.X509Certificates.OpenFlags]::ReadOnly)

$cert = $SourceStore.Certificates | Where-Object -FilterScript {
    $_.subject -like '*'
}

$DestStoreScope = 'LocalMachine'
$DestStoreName = 'My'

$DestStore = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Store -ArgumentList $DestStoreName, $DestStoreScope
$DestStore.Open([System.Security.Cryptography.X509Certificates.OpenFlags]::ReadWrite)
$DestStore.Add($cert)

$SourceStore.Close()
$DestStore.Close()

winrm create winrm/config/listener?Address=*+Transport=HTTPS `@`{CertificateThumbprint=`"($cert.Thumbprint)`"`}

Restart-Service winrm

download salt windows minion installer to the below path on the salt-master:
/salt/minion/, exe can be downloaded from https://docs.saltproject.io/en/latest/topics/installation/windows.html

wget https://repo.saltstack.com/windows/Salt-Minion-3003-Py3-AMD64-Setup.exe

Deploy Windows VM via salt:

salt-cloud -p w16k w16k-salty-minion -l debug

Deployed VM you can see firewall and salt minion installed:

Troubleshooting:

[ERROR   ] Unable to execute command
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 1005, in wait_for_psexecsvc
    stdout, stderr, ret_code = run_psexec_command(
  File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 956, in run_psexec_command
    client = Client(
  File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 879, in __init__
    self._client = PsExecClient(server, username, password, port, encrypt)
NameError: name 'PsExecClient' is not defined

cause: PsExecClient module is not installed. , use pip3 to install this

upgrade vRA 8.2 to 8.3

Upgrade LCM

Log in to LCM> Lifecycle operations> Settings> System upgrade

Create snapshot

Once snapshot is done, Click on Check for upgrade,

vRLCM will reboot as a part of the upgrade. Wait patiently… (go have a coffee)

On successfull upgrade, we should see :

Add vRA package repository to LCM

Download vRA 8.3 upgrade repository from https://my.vmware.com/group/vmware/downloads/details?downloadGroup=VRA-830&productId=1116&rPId=59213

copy the ISO to LCM, /data directory

Now, Go back to LCM> Lifecycle operations> settings > Binary mapping>Add product binaries, enter path in base location and then cick on discover

Clock on the prelude(the one we uploaded) and click on add

wait for the import task to compleate. You can take a look at the task by clicking on requests > Click on the most recent request

On successfull import, you should see it in the binary mapping:

Upgrade VRA:

Now that the upgrade packages have been uploaded to LCM, Go into LCM > LifeCycle Operations > Environments > Click on “view details” for the vRA environment.

Triger inventory sync:

Go grab a coffee!!, Typically at this point its just waiting for vRLCM to perform the upgrade.

Click on upgrade:

Run pre-check

wait for pre-check to complete: (takes about 3-10 min)

submit upgrade task

Now, Go grab a cup of coffee, This step generally takes a while.

Getting started with salt stack and VMware (vCenter)

Salt stack (salt open) is an opensource utility that can be used to orchestrate vCenter tasks. In this blog, I will guide you on how to install, and set a basic clone operation with salt stack.

Install:

pre-requisite: Linux VM (cent os or ubuntu), I’ve used Ubuntu 20.4 for the below example.

Log into ssh of the ubuntu VM, and add salt stack repository:

wget -O - https://repo.saltstack.com/py3/debian/10/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -

add key: in /etc/apt/sources.list.d/saltstack.list

nano /etc/apt/sources.list.d/saltstack.list

add the below line on the file:

deb http://repo.saltstack.com/py3/debian/10/amd64/latest buster main

update repository

sudo apt update

install stalt stack

sudo apt-get install salt-master salt-minion salt-ssh salt-syndic salt-cloud salt-api -y

Edit vi /etc/salt/master and add the salt master IP address to interface

interface: 172.16.8.9

Configuring VMware provider

Pre-requisite: Pyvomi must be installed

sudo apt install python3-pip
pip3 install pyVmomi

Confirm pyvomi is imported:

python3 -c "import pyVmomi" ; echo $?

Setup cloud provider:

create /etc/salt/cloud.providers.d/vmware.conf with the below contents: (replace user, password and url as per what you have in your enveronment)

#/etc/salt/cloud.providers.d/vmware.conf
vcsa:
 driver: vmware
 user: 'administrator@vs.lo'
 password: 'P@ssw0rd'
 url: 'vcsa.ntitta.lab'
 verify_ssl: False 

homelabs:
 driver: vmware
 user: 'administrator@vs.lo'
 password: '*************'
 url: 'vcsa.ntitta.in'

now you can test the above config by querying for images using below command:

salt-cloud --list-images vcsa

Okay, Now, lets set up profiles: Create /etc/salt/cloud.profiles.d/vmware.conf with the below content:

#/etc/salt/cloud.profiles.d/vmware.conf
ubuntu20:
 provider: vcsa
 clonefrom: ub20
 cluster: vSAN
 ssh_username: root
 password: 'P@ssw0rd'

update salt config

 salt-cloud -u

Test a deployment:

salt-cloud -p ubuntu20 test-stal1-vm

Troubleshooting: (enable debug by using the below )

salt-cloud -p w16k salty-w16-test-1 -l debug

certificate of FQDN has expired when attempting to add a product into the usage meter, specifically when using Sectigo signed certificates.

When attempting to add a product to usage meter, the product migth fail to add if it has a certificate signed by sectigo

Cause: https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020

Resolution: Import root certificates to the appliance java keystone.

Steps:
* take a snapshot of the appliance
* ssh to usage meter appliance
* change user to root

su root


* create or import the root certificate to the appliance

curl https://crt.sh/?d=1199354 >  /home/usagemeter/root.crt

Note: if you have a different CA provider, replace the below with the path to download the root certificate or simply scp the certificate to the UM appliance.

import the certificates (run the command as it is if the root is placed in /home/usagemeter/root.crt

keytool -import -trustcacerts -file /home/usagemeter/root.crt -alias USERTRUST -keystore /usr/java/jre-vmware/lib/security/cacerts

Note: Default keystore password is

changeit

on successfull import, you should see

now, go ahead and add the product back in to usage meter:

Note: when adding vCD, Please ensure that you add the endpoint in the format https://FQDN, IE: https://vcd.ntitta.in

Troubleshooting (show fill certificate chain, check the validity of the last certificate. ):

openssl s_client -showcerts -connect vcsa.ntitta.lab:443

vRA Deployments Fail with IPAM Provider Error: {“code”:3000,”message”:”The extensible attributes for search are not specified. (Dynamic Script Module name : findFixedAddressesByEA#27)”} (Dynamic Script Module name : throwIpamError#28)

After vRA was upgraded from 7.3 to 7.6 and the infoblox plugin to the latest, Deployments fail with the below message:

IPAM Provider Error: {"code":3000,"message":"The extensible attributes for search are not specified. (Dynamic Script Module name : findFixedAddressesByEA#27)"} (Dynamic Script Module name : throwIpamError#28)

After investigation from Infoblox, we found the Extensible attribute “VMware IPaddress” was missing on infoblox

Infoblox Documentation: https://docs.infoblox.com/download/attachments/8945695/Infoblox_IPAM_Plugin_for_VMware_vRA_User%27s_Guide.pdf?version=1&modificationDate=1606831097478&api=v2

In a second scenario, Looks like my DHCP range was fully allocated. we root caused this by reviewing the vRO workflow “allocate”

cannot verify certificate chain. when attempting to add vCenter to usage meter 4.x

  • bump up logs to trace and re-adding shows the below:
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager55 | Sending package: sendDataId=435014645 idx1=0, length=187
[2020-09-24 09:59:31]  | TRACE |   Gnats MsgProcessor | tion.GNatsConnection.GNatsMessageHandler | Get a packet from gnats magicId=1974333149, sendDataId=942819630, idx1=0, totalLen=76
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | tion.GNatsConnection.GNatsMessageHandler | GNats 'gateway_cli' processing message '{"authorization":"auth_stab","data":[],"errCode":"OK","respond_id":"id_140"}' from 'gateway_cli.cl
ient.responds' with reply 'null'
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | ection.client.RequestManager.RequestInfo | Respond for trackingID - id_140  => errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: -------- Internal API call ------
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: command - vrni
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: action - read
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: request data - {"productType":"VRNI"}
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 | re.um.umconnection.client.SendDataClient | ProductManager56 | sendData: trackingID - ProductManager56
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager56 | Gnats connection will send the data in 1 number of packages, id = 435014646, totalSize=186
[2020-09-24 09:59:31]  | TRACE | inx-clojure-worker-1 | m.vmware.um.umconnection.GNatsConnection | ProductManager56 | Sending package: sendDataId=435014646 idx1=0, length=186
[2020-09-24 09:59:31]  | TRACE |   Gnats MsgProcessor | tion.GNatsConnection.GNatsMessageHandler | Get a packet from gnats magicId=1974333149, sendDataId=942819631, idx1=0, totalLen=76
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | tion.GNatsConnection.GNatsMessageHandler | GNats 'gateway_cli' processing message '{"authorization":"auth_stab","data":[],"errCode":"OK","respond_id":"id_141"}' from 'gateway_cli.cl
ient.responds' with reply 'null'
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 |    com.vmware.um.umconnection.UmResponse | Responding with: errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG |     pool-2-thread-11 | ection.client.RequestManager.RequestInfo | Respond for trackingID - id_141  => errCode=OK errMsg=null errData=null JSonObj=null JSonArr=[]
[2020-09-24 09:59:31]  | DEBUG | inx-clojure-worker-1 |    com.vmware.um.umconnection.UmResponse | ProductManager56 | Responding with: errCode=ERR_PM_WRONG_HOST_NAME errMsg=Certificate error for vcenter.abc.ee: Can not verify certificate chain errData=null JSonObj=null JSonArr=null

when looking at the certificate of the venter

openssl s_client -connect vcenter.abc.ee:443
Certificate chain
 0 s:/CN=vcenter.abc.ee
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
 2 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
...
...
...
-----END CERTIFICATE-----
subject=/CN=vcenter.abc.ee
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
---

Cause: Incorrect ordering of the certificates.

if you look closely on the certificate chain:
Certificate 0 is for the vcenter server and was issued by RapidSSL. Certificate 1 is the DigiCert root certificate. And certificate 2 is the RapidSSL certificate, issued by DigiCert.

Apparently, web servers are often forgiving of this kind of out-of-order certificate chain, but it does violate the SSL spec. Because certificate 0 is signed by RapidSSL, certificate 1 needs to be the RapidSSL certificate, which is currently certificate 2 instead.

To resolve this, re-import the custom certificate to vCenter server with the chain in the correct order.

ie:

 Certificate chain
 0 s:/CN=vcenter.abc.ee
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018 
  1  s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL RSA CA 2018
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA  
 2  s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA 

Senario 2: You receive the error when attempting to add a product with custom certs or CA Signed certs.

vCenter:

For instance, if you are trying to add vCenter with CA/Custom root certificates, review the certificate chain imported on vCenter.

on the vCenter, run the below to review the certificate chain.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT

The certificate chain here must be in the correct order, Ie

-----BEGIN CERTIFICATE-----
signed certificate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
intermediate/subordinate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
intermediate/subordinate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
root
-----END CERTIFICATE-----

vCD

Review vCD certificates:

Keytool

vRA 8.x GitLab Integration

Create the GitLab integration in vRealize Automation:
1. Navigate to Infrastructure > Connections > Integrations.
2. Click + ADD INTEGRATION.
3. Select GitLab.
4. Enter the Name and Description.
5. Enter the server URL.
6. Enter the access token.
7. Click VALIDATE.
8. Click ADD.


The Access Token is copied from the GitLab console.

Preparing the GitLab Repository

You must create and save your blueprints in a specific structure in the GitLab repository so that vRealize Automation can detect them. The following prerequisites must be met:
• Create separate directories in the repository for each blueprint.
• Each directory must have one blueprint with the blueprint.yaml name.
• The blueprint must include the following properties at the top of the YAML code:

— name:
— version:

Creating Dedicated Directories

In the GitLab console, navigate to Repository and click New directory. Specify the directory name and description.
To have 10 blueprints source controlled in a vRealize Automation project, you must create 10 separate directories.

Naming the Blueprints

You must meet the following requirements:
• Blueprint Name:
— Blueprint name must be blueprint.yaml.
— Name is case-sensitive.
— Only one blueprint exists in a directory.
• Blueprint Properties:
— First property must be name:.
— Second property must be version:.

If you fail to meet these requirements of blueprint name and properties in the GitLab repository, the blueprints are not detected in vRealize Automation.
Each repository creates a default branch called master. You can create additional branches. The branch name is used when you add a content source in vRealize Automation.

Creating the GitLab Content Source

Add a content source to the GitLab connection to import blueprints from the specified repository.

To add a content source to the GitLab integration
1. Select Infrastructure > Integrations and select the GitLab integration.
2. Select Projects.
3. Select an existing vRealize Automation project.
4. Select if you want to import Blueprints or ABX action scripts.
5. Enter the Repository path in GitLab. The path is the user name of the main GitLab account appended to the GitLab project or repository name.
6. Enter the GitLab branch to use.
7. (Optional) Enter a folder (directory) name. If the folder name is blank, all GitLab directories are available. When you click ADD, an automated synchronization task is initiated that imports blueprints into vRealize Automation. When the synchronization tasks are complete, a message indicates that the blueprints are imported.

vRealize Automation and GitLab Projects

The mapping between projects in GitLab and projects in vRealize Automation is one-to-one:
• For example, if you want to source control your blueprints for three projects in vRealize Automation, you must create three projects in GitLab.
• All the blueprints in vRealize Automation are tied with a vRealize Automation project.
• Selecting a vRealize Automation project is mandatory to create a GitLab content source.
• Selecting an existing vRealize Automation project (content source) with a new GitLab project results in a failure

Verifying the Blueprint Synchronization

You can verify the following blueprint synchronization:
• GitLab Integration: Project synchronization status
• GitLab Integration: Project synchronization history
• Cloud Assembly Design Tab: Imported blueprints

vRA 7.6 HP upgrade fails RPM already installed

vRA 7.6 HP upgrade fails with

2020-11-11T11:03:19.964428+01:00 srv01vraapp2t.corp.trumpf.com vcac-config: INFO  com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.isAllCommandExecuted:1085 - Checking if all commands a
re executed
2020-11-11T11:03:20.305569+01:00 srv01vraapp2t.corp.trumpf.com vcac-config: ERROR com.vmware.vcac.configuration.utils.ProcessUtil.execute:22 - Command /bin/sh returned invalid status 8. Output: Preparing...
                ##################################################
, Error:        package horizon-service-rpm-3.1.0.0-15448541.noarch (which is newer than horizon-service-rpm-3.1.0.0-15433743.noarch) is already installed
2020-11-11T11:03:20.305569+01:00 srv01vraapp2t.corp.trumpf.com vcac-config: ERROR com.vmware.vcac.cli.configurator.ConfiguratorImpl.processExceptionResult:160 - Command execution failed with unexpected erro
r: com.vmware.vcac.configuration.utils.ProcessUtil$ProcessExecutionException: package horizon-service-rpm-3.1.0.0-15448541.noarch (which is newer than horizon-service-rpm-3.1.0.0-15433743.noarch) is already
 installed.
com.vmware.vcac.configuration.utils.ProcessUtil$ProcessExecutionException: package horizon-service-rpm-3.1.0.0-15448541.noarch (which is newer than horizon-service-rpm-3.1.0.0-15433743.noarch) is already in
stalled
        at com.vmware.vcac.configuration.utils.ProcessUtil.execute(ProcessUtil.java:23) ~[vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
        at com.vmware.vcac.configuration.utils.ProcessUtil.executeShellCommand(ProcessUtil.java:96) ~[vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
        at com.vmware.vcac.cli.configurator.commands.cluster.patch.ExecuteShellCommand.execute(ExecuteShellCommand.java:28) ~[vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
        at com.vmware.vcac.cli.configurator.commands.cluster.ClusterPatchCommand.execute(ClusterPatchCommand.java:53) ~[vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
        at com.vmware.vcac.cli.configurator.ConfiguratorImpl.execute(ConfiguratorImpl.java:109) [vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
        at com.vmware.vcac.cli.configurator.Configurator.main(Configurator.java:123) [vcac-config-cli-7.6.0-SNAPSHOT.jar:?]
2020-11-11T11:03:29.144546+01:00 srv01vraapp2t [database-failover-agent][3932]: 2020/11/11 11:03:29 --- AGENT next iteration ---
2020-11-11T11:03:29.145101+01:00 srv01vraapp2t [database-failover-agent][3932]: 2020/11/11 11:03:29 getAllVotes():, url suffix: api/master
2020-11-11T11:03:30.145235+01:00 srv01vraapp2t [database-failover-agent][3932]: 2020/11/11 11:03:30 ElectMaster(): Votes for Master:
Node localhost (On: true, Manual failover: false, IsLocalDbMaster: true) has 1 voters: localhost

Resolution

Run

  "rpm -Uvh --replacepkgs /usr/lib/vcac/patches/repo/cafe/patchRpms/*.rpm", --oldPackage 

and then re-try the upgrade.

if the problem persists then

del /usr/lib/vcac/patches/repo/cafe/patchRpms/horizon-service-rpm-3.1.0.0-15433743.noarch.rpm


and then re-run the upgrade