NSLCD based auth on Linux machine/Usage meter 4.4 LDAP

Below is the sample configuration to get UM-Ad intigation via LDAP (replace the contents of /ec/nslcd.conf with what is appropriate in your environment.

root@photon-machine [ /etc ]# cat /etc/nslcd.conf


uid nslcd
gid ldap

uri ldap://ntitta.lab
base dc=ntitta,dc=lab
binddn CN=service,CN=Users,DC=ntitta,DC=lab
bindpw P@ssw0rd

pagesize 1000
filter passwd (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
map    passwd uid              sAMAccountName
map    passwd homeDirectory    unixHomeDirectory
map    passwd gecos            displayName
filter shadow (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
map    shadow uid              sAMAccountName
map    shadow shadowLastChange pwdLastSet
filter group  (objectClass=group)

cat /etc/nsswitch.conf


hosts: files resolve dns
networks: files

protocols: files
services: files
ethers: files
rpc: files


passwd: files ldap
group: files ldap
shadow: files ldap

cat /etc/pam.d/system-auth

# Begin /etc/pam.d/system-auth
auth    required    pam_env.so
auth    required    pam_tally2.so onerr=fail deny=3 unlock_time=900 root_unlock_time=900 file=/var/log/tallylog
auth    sufficient  pam_ldap.so
auth    required    pam_unix.so
auth    optional    pam_faildelay.so delay=4000000

cat /etc/pam.d/system-account

# Begin /etc/pam.d/system-account
account    sufficient  pam_ldap.so
account    required    pam_tally2.so file=/var/log/tallylog
account    required    pam_unix.so
# End /etc/pam.d/system-account

cat /etc/pam.d/system-password

# Begin /etc/pam.d/system-password
password    sufficient  pam_ldap.so try_first_pass
password    requisite   pam_cracklib.so     minlen=10 minclass=4 difok=4 maxsequence=0 retry=3 enforce_for_root
password    requisite   pam_pwhistory.so    retry=3 remember=5 enforce_for_root
password    required    pam_unix.so         sha512 shadow use_authtok
# End /etc/pam.d/system-password

cat /etc/pam.d/system-session

# Begin /etc/pam.d/system-session
session   required    pam_unix.so
session   required    pam_limits.so
session   optional    pam_motd.so
session   optional    pam_lastlog.so showfailed
session   optional    pam_systemd.so
session   optional    pam_ldap.so
# End /etc/pam.d/system-session

cat /etc/pam.d/vmware-um-pam

auth       sufficient /lib64/security/pam_ldap.so
auth       required   /lib64/security/pam_unix_auth.so
account    sufficient /lib64/security/pam_ldap.so
account    required   /lib64/security/pam_unix_acct.so

aside from this, NSLCD also needs some pre-requisites for the user that is used to log in, This is described here: https://www.server-world.info/en/note?os=Windows_Server_2019&p=active_directory&f=12
basically the below attributes must exist on ad for the user in question:

uid           sAMAccountName
uidNumber     objectSid:<yourValue>
gidNumber     primaryGroupID
homeDirectory "/home/$sAMAccountName"
gecos         displayName
loginShell    "/bin/bash"
gidNumber      primaryGroupID

Note:

  • uid must be uniq for a user and should not be associated with any existing user in UM appliance, see getent passwd for the full list of used uids
  • gid must exist in the UM appliance before configuring AD. Use it to control what kind of privileges you want to give to the user. In most case use the usgemeger gid 1002

Note: Making changes to /etc/pam.d generally needs a reboot to take effect. Once the above config are in place, reboot the appliance and try login in
Note: If you move the original config files and create new files, please ensure the permissions of the files are corrected.

If configured correctly and when the user is attempted to log in on the UI, you should see the same domain user mapped in getent passwd

root@photon-machine [ /etc ]# getent passwd


root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/dev/null:/bin/false
daemon:x:6:6:Daemon User:/dev/null:/bin/false
messagebus:x:18:18:D-Bus Message Daemon User:/var/run/dbus:/bin/false
systemd-bus-proxy:x:72:72:systemd Bus Proxy:/:/bin/false
systemd-journal-gateway:x:73:73:systemd Journal Gateway:/:/bin/false
systemd-journal-remote:x:74:74:systemd Journal Remote:/:/bin/false
systemd-journal-upload:x:75:75:systemd Journal Upload:/:/bin/false
systemd-network:x:76:76:systemd Network Management:/:/bin/false
systemd-resolve:x:77:77:systemd Resolver:/:/bin/false
systemd-timesync:x:78:78:systemd Time Synchronization:/:/bin/false
nobody:x:65534:65533:Unprivileged User:/dev/null:/bin/false
sshd:x:50:50:sshd PrivSep:/var/lib/sshd:/bin/false
named:x:999:999::/var/lib/bind:/bin/false
polkitd:x:27:1000:PolicyKit Daemon Owner:/etc/polkit-1:/bin/false
nslcd:x:998:998:nslcd ldap user:/:/usr/sbin/nologin
ntp:x:87:87:Network Time Protocol:/var/lib/ntp:/bin/false
usagemeter:x:1000:1002::/home/usagemeter:/bin/bash
umauditor:x:1001:1003::/home/umauditor:/bin/bash
test:*:5000:1002:test:/home/test:/bin/bash                         <---------this is the domain user

vRA patching: clearing the patch tab from a failed patch

Symptoms: vRA patching failed and is now stuck with a patch in the repository, the remove button does not do anything, the retry button is grayed out:

Note: Its always recommended that you take a powered off snapshot of all the nodes before patching and before performing the below:

  • Take a powered off  snapshot of all the vRA nodes and the IAAS nodes, Take an IAAS DB (sql db backup) (power everything down and take a snapshot, not a rolling power off)
  • Power them back up in order, once the services are up and registered proceed with the below: 
  • on every vcac (vra) nodes, Delete the contents of /usr/lib/vcac/patches folder (rm -rf /usr/lib/vcac/patches/*)
  • Check if the file “/opt/vmware/share/htdocs/service/cafe/patch_upload.lock” is present, if yes delete.
  • go back to vami and confirm if it allows uploading the patch, upload and then patch

Clean up vcac DB:

su postgres
psql 
/c vcac

delete from hf_execution_cmd;
delete from hf_patch_execution;

delete from hf_patch_nodes;
delete from hf_patch;

Usage Meter 4.x picks up vRops although vRops no longer exist on the environment

usage meter 4.x picks up vRops from vCenter mob. after adding the vCenter , UM does dynamic discovery.  if for any reason, vRops extension is left stale behind on vCenter, UM will continue to think the product exists. 

To resolve this,
log in to the vCenter mob and remove com.vmware.vrops and com.vmware.vcops 

Instructions to remove the extensions can be found here: https://kb.vmware.com/s/article/1025360 , Kindly ensure that you remove only com.vmware.vrops and com.vmware.vcops.

Once the extension is removed, remove and re-add the product: vCenter (all vCenter’s referenced by this vRops)  back on to usage meter. you should no longer see vRops. 

adding a webhook to vRA-Code Stream fails

adding a webhook to vRA-Code Stream fails with error:

java.lang.IllegalArgumentException: 400 BAD_REQUEST "Unable to create webhook at Git server. Request failed with : 422"

browser console:

{timestamp: 1622639483332, path: "/codestream/api/git-webhooks", status: 400, error: "Bad Request",…}
@type: "com.vmware.codestream.exception.CodestreamException"
error: "Bad Request"
message: "400 BAD_REQUEST \"Unable to create webhook at Git server. Request failed with : 422\""
path: "/codestream/api/git-webhooks"
requestId: "929b7098-102300"
status: 400
timestamp: 1622639483332

Symptoms: Gitlab is installed on the same network as of vRA/Codestream


Cause: Gitlab does not allow connections to create a webhook from local subnet by default.

Resolution :Allow requests to the local network from web hooks and services

Steps:

  • On gitlab, go to admin area:
  • On the left hand, navigate to settings> Network
  • enable Allow requests to the local network from system hooks, also add the codestream FQDN (or *) in the URL

Now Go back to codestream and add the webhook: success!!

vcsa update “stage only” and “stage and install grayed out”

screenshot:

Solution: remove/rename the software update status config file

mv /etc/applmgmt/appliance/software_update_state.conf /etc/applmgmt/appliance/software_update_state.conf.bak

On a side note: I ran into the issue after attempting to “stage and install” my homelabs vcsa,
logs reveal:

/var/log/vmware/applmgmt/applmgmt.log
2021-05-30T15:20:26.108 [3758]DEBUG:vmware.appliance.update.update_functions:Removing the mount point /mnt/iso-contents
2021-05-30T15:20:26.109 [3758]INFO:vmware.appliance.update.update_functions:ISO unmounted successfully
2021-05-30T15:20:26.109 [3758]DEBUG:vmware.appliance.update.update_b2b:discoverLocalUpdate failed.
Traceback (most recent call last):
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1434, in _discoverUpdateAt
    tempFolder)
  File "/usr/lib/python3.7/shutil.py", line 248, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/iso-contents/manifest-latest.xml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1640, in discoverLocalUpdatesNoException
    _discoverLocalUpdates()
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1631, in _discoverLocalUpdates
    _discoverUpdateAtIso()
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1607, in _discoverUpdateAtIso
    raise e
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1602, in _discoverUpdateAtIso
    _discoverUpdateAt(manifestDir, packagesDir, copyFileFunc, 'iso')
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_b2b.py", line 1446, in _discoverUpdateAt
    raise RpmManifestNotFoundException
vmware.appliance.update.update_b2b.RpmManifestNotFoundException

so, I did the above, staged the upgrade repository this time:

Then ran the install:


Progress:

vrbc 7.6 security patch (7.6.0.46000) installation fails

installing vRBC security patch fails:

logs:

/opt/vmware/var/log/vami/vami.log
/opt/vmware/var/log/vami/updatecli.log

pg_dump: dumping contents of table "itfm_cloud_admin.vra_vm_details"
pg_dump: dumping contents of table "itfm_cloud_admin.vra_vm_details_tags"
pg_dump: saving large objects
Dump successful
Stopping VMware vPostgres: ok
MongoDB instance is already upgraded. Aborting upgrade
error: package libyui-ncurses-pkg6-2.46.1-3.4.x86_64 is not installed
error: package perl-Bootloader-YAML is not installed
30/05/2021 7:30:50 [INFO] Update status: Done pre-install scripts
30/05/2021 7:30:50 [INFO] Update status: Running installation tests
30/05/2021 7:30:50 [INFO] Running /opt/vmware/var/lib/vami/update/data/job/15/test_command 
Preparing packages...
	installing package kernel-default-4.12.14-122.26.1.x86_64 needs 4MB on the /boot filesystem
30/05/2021 7:30:51 [ERROR] Failed with exit code 65024
30/05/2021 7:30:51 [INFO] Update status: Running post-install scripts

Cause: the /boot partition ran out of space

Resolution:

Take a snapshot of the vrb appliance and perform the below:

  • SSH to vRB VA.
  • Create temp directory and move old kernel files.
mkdir /tmp/boot
cd /boot/
mv vmlinu* initrd* /tmp/boot
  • re-run the upgrade to the security build

Usage meter 4.3/4.4 vCenter server Partial collection failure: Events

Usage meter reports partial collection failure: events

Logs files: 	vccol_main.log | vccol_error.log
[2021-05-18 09:02:25]  | ERROR | ter collector thread |    com.vmware.um.vccollector.VCCollector | vCenter collector90 | Events stage raised exception javax.xml.ws.WebServiceException: java.net.SocketTimeoutException: Read timed out java.net.SocketTimeoutException: Read timed out=>Read timed out
[2021-05-18 09:02:26]  | ERROR | ter collector thread | com.vmware.um.collector.CollectionHelper | vCenter collector98 | Status (COLLECT_API_ERR) for vCenter server 7: Partial collection failure: Events
[2021-05-18 10:02:18]  | ERROR | ter collector thread |    com.vmware.um.vccollector.VCCollector | vCenter collector179 | Events stage raised exception javax.xml.ws.WebServiceException: java.net.SocketTimeoutException: Read timed out java.net.SocketTimeoutException: Read timed out=>Read timed out

Cause: Connection was closed before the data could be retrieved successfully. Usage meter requests vCenter for events, this api generally takes some time to respond either due to the huge number of events or slowly due to heavy processing on the vCenter.

Resolution: Increase timeouts
* take a snapshot of the um appliance.
* ssh into the appliance with the user usagemeter
* take a backup copy of common_utils.sh

cp /opt/vmware/cloudusagemetering/scripts/common_utils.sh /opt/vmware/cloudusagemetering/scripts/common_utils.sh.bak

* edit the config file using vi

 vi /opt/vmware/cloudusagemetering/scripts/common_utils.sh

* replace the values of the below field

=> CONNECT_TIMEOUT_MS="300000"
=> READ_TIMEOUT_MS="600000"

* save the file and restart the appliance


Note: On the vCenter side, if there are bursts of events, then this is also a likely scenario. KB https://kb.vmware.com/s/article/74607 is one among several where the burst is documented (event bursts need to be triaged from vCenter prospective)

Saltstack + vSphere: Deploying Windows VM’s with Windows Minion

Ensure that you have set up sphere provider provider, refer my previous blog https://blog.ntitta.in/?p=597

create a windows profile

/etc/salt/cloud.profiles.d/w16k.conf 

root@saltyub:/# cat /etc/salt/cloud.profiles.d/w16k.conf 
w16k:
  provider: vcsa
  clonefrom: w16k_salt 
#  devices: 
#   network: 
#    Network adaptor 1:
#     name: VM Network
#     adapter_type: vmxnet3
#     switch_type: standard
#     ip: 172.16.70.79
#     gateway: [172.16.1.1]
#     subnet_mask: 255.255.128.0
#     domain: ntitta.lab
  cluster: vSAN
  datastore: vsanDatastore
  power_on: True
  deploy: True
  customization: True
  minion:
   master: saltyu.ntitta.lab
  win_username: administrator 
  win_password: 'P@ssw0d'
  plain_text: True
  win_user_fullname: admin
  win_run_once: 'powershell.exe c:\scripts\e.winrm.ps1'
  win_installer: /salt/minion/Salt-Minion-3000.9-Py2-AMD64-Setup.exe
  winrm_verify_ssl: False

Ensure that you have the smbprotocol and pypsexec installed

pip3 install smbprotocol
pip3 install pypsexec

on the guest windows server template, ensure vmware tools is installed and create a PowerShell script in the path: c:\scripts\e.winrm.ps1, refer salt doc for more information: https://docs.saltproject.io/en/latest/topics/cloud/windows.html

New-NetFirewallRule -Name "SMB445" -DisplayName "SMB445" -Protocol TCP -LocalPort 445
New-NetFirewallRule -Name "WINRM5986" -DisplayName "WINRM5986" -Protocol TCP -LocalPort 5986

winrm quickconfig -q
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="300"}'
winrm set winrm/config '@{MaxTimeoutms="1800000"}'
winrm set winrm/config/service/auth '@{Basic="true"}'

$SourceStoreScope = 'LocalMachine'
$SourceStorename = 'Remote Desktop'

$SourceStore = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Store -ArgumentList $SourceStorename, $SourceStoreScope
$SourceStore.Open([System.Security.Cryptography.X509Certificates.OpenFlags]::ReadOnly)

$cert = $SourceStore.Certificates | Where-Object -FilterScript {
    $_.subject -like '*'
}

$DestStoreScope = 'LocalMachine'
$DestStoreName = 'My'

$DestStore = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Store -ArgumentList $DestStoreName, $DestStoreScope
$DestStore.Open([System.Security.Cryptography.X509Certificates.OpenFlags]::ReadWrite)
$DestStore.Add($cert)

$SourceStore.Close()
$DestStore.Close()

winrm create winrm/config/listener?Address=*+Transport=HTTPS `@`{CertificateThumbprint=`"($cert.Thumbprint)`"`}

Restart-Service winrm

download salt windows minion installer to the below path on the salt-master:
/salt/minion/, exe can be downloaded from https://docs.saltproject.io/en/latest/topics/installation/windows.html

wget https://repo.saltstack.com/windows/Salt-Minion-3003-Py3-AMD64-Setup.exe

Deploy Windows VM via salt:

salt-cloud -p w16k w16k-salty-minion -l debug

Deployed VM you can see firewall and salt minion installed:

Troubleshooting:

[ERROR   ] Unable to execute command
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 1005, in wait_for_psexecsvc
    stdout, stderr, ret_code = run_psexec_command(
  File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 956, in run_psexec_command
    client = Client(
  File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 879, in __init__
    self._client = PsExecClient(server, username, password, port, encrypt)
NameError: name 'PsExecClient' is not defined

cause: PsExecClient module is not installed. , use pip3 to install this

adding vRA 7.6 to usage meter 4.3 fails with invalid credentials

Logs:

[2021-03-23 15:02:49]  | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 15. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED

[2021-03-23 15:02:57] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:02:57] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 15. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:03:21] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:03:21] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 15. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:14:17] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:14:17] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 15. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:19:50] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:19:50] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 16. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:47:09] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED
[2021-03-23 15:47:09] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 16. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: stream was reset: HTTP_1_1_REQUIRED=>stream was reset: HTTP_1_1_REQUIRED

Resolution: Disable http2 on the IIS and then re-add the product



Instructions:
* Start → regedit
* Navigate to the folder/path: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters
* Under the Parameters folder, right-click the white-space, add 2 new DWORD (32-bit) values:
 – EnableHttp2Tls
 – EnableHttp2Cleartext
* Ensure both new values have been set to 0(disabled) by right-clicking the value and clicking “Modify…”
* Restart the OS.



Misc: To test iis credentials, you can perform the below curl command (run it from a different linux instance as UM has enforced FIPS which disables MD5 for ntlm based authentication)

curl --ntlm --user 'Administrator':'VMware1!' https://iis.ntitta.lab/repository/Data/ManagementModelEntities.svc/ProvisioningGroups -k



Once the above is done, you are likely to run into a second issue, Logs:

[2021-03-26 15:22:01]  |  WARN | t_credentials_create | vmware.um.common.http.LoggingInterceptor | GET /repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed with Too many follow-up requests: 21 Too many follow-up requests: 21

[2021-03-26 15:22:01] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: Too many follow-up requests: 21
[2021-03-26 15:22:01] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | IaaS credentials error for server 5. Please correct the IaaS credentials. GET https://iis.ntitta.lab:443/repository/Data/ManagementModelEntities.svc/ProvisioningGroups failed: Too many follow-up requests: 21=>Too many follow-up requests: 21

To resolve the above, Edit the vRA credentials in usage meter and under the IIS credentials, write the user name in the format: user@domain.com and leave the domain field blank

This image has an empty alt attribute; its file name is image-1024x808.png

if you see the below:

[2021-03-29 11:29:03]  |  INFO | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | Start test credentials for vRA Cafe server '16'

[2021-03-29 11:29:03] | ERROR | t_credentials_create | com.vmware.um.vracollector.api.VRAClient | Server '16' failed to authenticate API endpoint identity/api/tokens on server 16 returned HTTP status 400

this means that the vRA credentials are incorrect. Try setting the user name as : administrator and then save.