Duplicate STS signing certificates can cause authentication failure.

Log:

04-15T23:27:57.946Z | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.sso.client.exception.TimeSynchronizationException: Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Mon Apr 15 23:27:57 UTC 2019, endTime=Tue Apr 16 01:07:57 UTC 2019] :: Signing certificate is not valid at Mon Apr 15 23:27:57 UTC 2019, cert validity: TimePeriod [startTime=Wed Apr 11 18:17:03 UTC 2018, endTime=Thu Apr 11 18:17:03 UTC 2019]
        at com.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:182)
        at com.vmware.vapi.endpoint.cis.StsBuilder.rebuild(StsBuilder.java:77)
        at com.vmware.vapi.endpoint.cis.StsBuilder.buildInitial(StsBuilder.java:54)
        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.sso.client.exception.TimeSynchronizationException: Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Mon Apr 15 23:27:57 UTC 2019, endTime=Tue Apr 16 01:07:57 UTC 2019] :: Signing certificate is not valid at Mon Apr 15 23:27:57 UTC 2019, cert validity: TimePeriod [startTime=Wed Apr 11 18:17:03 UTC 2018, endTime=Thu Apr 11 18:17:03 UTC 2019]
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition(SecurityTokenServiceImpl.java:1016)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:932)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:856)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:477)
        at com.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:179)
        ... 12 more

The duplicate certificate needs to be deleted via jxplorer, refer to the screenshot above (from the above example, trustedcertchain-2 and trusted credential-2 had the same certificate which had to be deleted from the system)

Handy commands when working with vPostgres

Copy Table to CSV

copy vpx_vm to '/tmp/vpx_vm' DELIMITER ',' CSV;

Copy table from CSV

copy table FROM '/tmp/table.csv' DELIMITER ',' CSV;

list Top tables by size:

SELECT
  schema_name,
  relname,
  pg_size_pretty(table_size) AS size,
  table_size

FROM (
       SELECT
         pg_catalog.pg_namespace.nspname           AS schema_name,
         relname,
         pg_relation_size(pg_catalog.pg_class.oid) AS table_size

       FROM pg_catalog.pg_class
         JOIN pg_catalog.pg_namespace ON relnamespace = pg_catalog.pg_namespace.oid
     ) t
WHERE schema_name NOT LIKE 'pg_%'
ORDER BY table_size DESC;

postgres password:

root@is-dhcp34-161 [ / ]# cat ~/.pgpass

localhost:5432:replication:replicator:*v&w1pTkmZY}Q2<z
127.0.0.1:5432:replication:replicator:*v&w1pTkmZY}Q2<z
/var/run/vpostgres:5432:replication:replicator:*v&w1pTkmZY}Q2<z
localhost:5432:postgres:postgres:_ouG|OZ4NUwna0fB
127.0.0.1:5432:postgres:postgres:_ouG|OZ4NUwna0fB
localhost:5432:VCDB:postgres:_ouG|OZ4NUwna0fB
127.0.0.1:5432:VCDB:postgres:_ouG|OZ4NUwna0fB
/var/run/vpostgres:5432:VCDB:postgres:_ouG|OZ4NUwna0fB
/var/run/vpostgres:5432:postgres:postgres:_ouG|OZ4NUwna0f

Rename table

ALTER TABLE tablename RENAME TO new_table;

Database backup/Dump using pg_dump

pg_dump  VCDB -U postgres > /tmp/dump

Database backup (excluding a specific corrupted table)

pg_dump  VCDB -U postgres  -T vpx_host > /tmp/dump.excluting.currupt.table

Note: T: Exclude table and grab the rest.
      t: backup specific table only.

Determining broken tables(pg toast)

 for ((i=0; i<"668"; i++ )); do /opt/vmware/vpostgres/current/bin/psql -U "postgres" "VCDB" -c "SELECT * FROM VPX_VM LIMIT 1 offset $i" >/dev/null || echo $i; done

Note: Replace 668 with the highest table count on your setup

Starting postgres in single user mode:

su - postgres –c postgres --single -D /storage/db/vpostgres

use below to fix checksum on start 
su - postgres –c /path_to_postgres/postgres --single -D /storage/db/vpostgres -c fix_block_checksum="1663/1636/1694/0/978"

Note: Postgres will only start as the user postgres. ignoring the su - postgres will likely cause the service to fail

Reset WAL (Write Ahead logs)

pg_resetxlog /storage/db/vpostgres

Error “MultiXactId has not been created yet — apparent wraparound”

VACUUM FREEZE table_name; 

Manual Recovery of corrupt records from a postgres table.

A bash while loop that creates an SQL file to export every row individually into individual files. The rows with corrupt records are skipped.
you will need a unique readable collum. For the below example I have exported surr_id from vpx_text_array as /tmp/SURR_KEY_vpx_text_array and use a while loop to create individual lines for export on the .SQL file


while read p; do  echo "copy (select * from vpx_text_array where surr_key=$p) to '/db/$p' delimiter ',' csv;" >> /db/out.sql; done < /tmp/SURR_KEY_vpx_text_array

Use psql commandline to invoke the SQL script

/opt/vmware/vpostgres/9.4/bin/psql -U postgres -d VCDB -a -f /db/out.sql >> /db.export.err
psql.bin:/db/out.sql:153: ERROR:  missing chunk number 0 for toast value 66095 in pg_toast_19389  <----currupt records. TBR from other tables
psql.bin:/db/out.sql:190: ERROR:  missing chunk number 0 for toast value 66096 in pg_toast_19389  <----currupt records. TBR from other tables

Query to list top events from the events DB.

SELECT COUNT(EVENT_ID) AS NUMEVENTS, EVENT_TYPE, USERNAME FROM VPXV_EVENT_ALL GROUP BY EVENT_TYPE, USERNAME ORDER BY NUMEVENTS DESC LIMIT 10; 
 

DB schema paths

VCDB: 
VUMDB: /usr/lib/vmware-updatemgr/share/VCI_base_postgresql.sql 

Replacing VMware-stsd certificate on 6.5/6.7

If the vCenter was upgraded from 5.5, it retains legacy endpoints for lookup service
IE: https://FQDN:7444/lookupservce/sdk
Running certificate-manager will not replace the certificate and vCenter might actually start to complain about expired certificate although they are valid.

In this can be easily worked around by replacing the certificate on the VECS store:

Start by exporting the STS_INTERNAL_SSL_CERT and MACHINE_SSL_CERT

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /ssl/machine_ssl.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /ssl/machine_ssl.key


/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output /ssl/STS_INTERNAL_SSL_CERT-__MACHINE_CERT.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output /ssl/STS_INTERNAL_SSL_CERT-__MACHINE_CERT.key

Delete the contents of STS_INTERNAL_SSL_CERT store

 /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT -y

import machine_ssl store to the STS store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert  /ssl/machine_ssl.crt --key /ssl/machine_ssl.key 

Restart services and confirm service status

VMCA Certificates replacement for vCenter 6.x, workflow

VMCA is the Default self-signed certificates that is set up at the time of vCenter deployment.

All solution users and machine SSL certificates are signed with this certificate.

VMCA certificates can be regenerated by using option 8 on the certificatae manager.

root@is-dhcp36-107 [ / ]# /usr/lib/vmware-vmca/bin/certificate-manager
                 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
                |                                                                     |
                |      *** Welcome to the vSphere 6.5 Certificate Manager  ***        |
                |                                                                     |
                |                   -- Select Operation --                            |
                |                                                                     |
                |      1. Replace Machine SSL certificate with Custom Certificate     |
                |                                                                     |
                |      2. Replace VMCA Root certificate with Custom Signing           |
                |         Certificate and replace all Certificates                    |
                |                                                                     |
                |      3. Replace Machine SSL certificate with VMCA Certificate       |
                |                                                                     |
                |      4. Regenerate a new VMCA Root Certificate and                  |
                |         replace all certificates                                    |
                |                                                                     |
                |      5. Replace Solution user certificates with                     |
                |         Custom Certificate                                          |
                |                                                                     |
                |      6. Replace Solution user certificates with VMCA certificates   |
                |                                                                     |
                |      7. Revert last performed operation by re-publishing old        |
                |         certificates                                                |
                |                                                                     |
                |      8. Reset all Certificates                                      |
                |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]:

choosing option 8, you are presented with the below options:

Option[1 to 8]: 8
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [[email protected]]:[email protected]
Enter password:

Please configure certool.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : US] :

Enter proper value for 'Name' [Default value : CA] :

Enter proper value for 'Organization' [Default value : VMware] :

Enter proper value for 'OrgUnit' [Default value : VMware Engineering] :

Enter proper value for 'State' [Default value : California] :

Enter proper value for 'Locality' [Default value : Palo Alto] :

Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] :

Enter proper value for 'Email' [Default value : [email protected]] :

Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : is-dhcp36-107.isl.vmware.com

Enter proper value for VMCA 'Name' :VMCA
Continue operation : Option[Y/N] ? : y

You are going to reset by regenerating Root Certificate and replace all certificates using VMCA
Continue operation : Option[Y/N] ? : y

Below are the .cfg’s created for the certificate

root@is-dhcp36-107 [ /var/tmp/vmware ]# ls -ltrh
total 32K
drwxr-xr-x 3 root root 4.0K Mar 14 00:07 cis-license
-rw-r--r-- 1 root root  191 Apr  2 19:24 certool.cfg
-rw-r--r-- 1 root root  243 Apr  2 19:24 vsphere-webclient.cfg
-rw-r--r-- 1 root root  240 Apr  2 19:24 vpxd-extension.cfg
-rw-r--r-- 1 root root  230 Apr  2 19:24 vpxd.cfg
-rw-r--r-- 1 root root   87 Apr  2 19:24 root.cfg
-rw-r--r-- 1 root root  217 Apr  2 19:24 MACHINE_SSL_CERT.cfg
-rw-r--r-- 1 root root  233 Apr  2 19:24 machine.cfg

Workflow (below are the commands that are run in the background)

/usr/lib/vmware-vmafd/bin/dir-cli service list --login [email protected] --password *****
1. machine-6368669f-591a-44fa-bfb3-a76b166bfed6
2. vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6
3. vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6
4. vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6

Create certificate cfg files for the respective services

  • root.cfg
root@is-dhcp36-107 [ /var/tmp/vmware ]# cat root.cfg
Country = US
Name = VMCA
OrgUnit = VMware Engineering
State = California
#IPAddress =
  • Solution users


root@is-dhcp36-107 [ /var/tmp/vmware ]# cat machine.cfg
Country = US
Name = machine-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = [email protected]
Hostname = is-dhcp36-107.isl.vmware.com


root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vsphere-webclient.cfg
Country = US
Name = vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = [email protected]
Hostname = is-dhcp36-107.isl.vmware.com


root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vpxd.cfg
Country = US
Name = vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = [email protected]
Hostname = is-dhcp36-107.isl.vmware.com



root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vpxd-extension.cfg
Country = US
Name = vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = [email protected]
Hostname = is-dhcp36-107.isl.vmware.com



  • Machine SSL.cfg
root@is-dhcp36-107 [ /var/tmp/vmware ]# cat MACHINE_SSL_CERT.cfg
Country = US
Name = is-dhcp36-107.isl.vmware.com
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = [email protected]
Hostname = is-dhcp36-107.isl.vmware.com

Check if a backup store exist.. if not, create one.

/usr/lib/vmware-vmafd/bin/vecs-cli store list
MACHINE_SSL_CERT
TRUSTED_ROOTS
TRUSTED_ROOT_CRLS
machine
vsphere-webclient
vpxd
vpxd-extension
SMS

/usr/lib/vmware-vmafd/bin/vecs-cli store create --name BACKUP_STORE
service-control --start vmafdd
service-control --start vmcad
service-control --start vmdird

Export exiting certificate and import them to the backup store

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/vmware/old_machine_ssl.crt
/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp___MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp___MACHINE_CERT --cert /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.crt --key /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store machine --alias machine --output /storage/certmanager/rollback/machine_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /storage/certmanager/rollback/machine_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_machine --cert /storage/certmanager/rollback/machine_bkp.crt --key /storage/certmanager/rollback/machine_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vsphere-webclient --alias vsphere-webclient --output /storage/certmanager/rollback/vsphere-webclient_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /storage/certmanager/rollback/vsphere-webclient_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vsphere-webclient --cert /storage/certmanager/rollback/vsphere-webclient_bkp.crt --key /storage/certmanager/rollback/vsphere-webclient_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd --alias vpxd --output /storage/certmanager/rollback/vpxd_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /storage/certmanager/rollback/vpxd_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vpxd --cert /storage/certmanager/rollback/vpxd_bkp.crt --key /storage/certmanager/rollback/vpxd_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /storage/certmanager/rollback/vpxd-extension_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /storage/certmanager/rollback/vpxd-extension_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vpxd-extension --cert /storage/certmanager/rollback/vpxd-extension_bkp.crt --key /storage/certmanager/rollback/vpxd-extension_bkp.priv

Generate Machine_SSL certificate

/usr/lib/vmware-vmca/bin/certool --getrootca --server localhost
/usr/lib/vmware-vmca/bin/certool --selfca --config /var/tmp/vmware/root.cfg --server localhost
/usr/lib/vmware-vmca/bin/certool --getrootca --server localhost
/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/MACHINE_SSL_CERT.priv --pubkey=/storage/certmanager/MACHINE_SSL_CERT.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/MACHINE_SSL_CERT.priv --cert=/storage/certmanager/MACHINE_SSL_CERT.crt --config=/var/tmp/vmware/MACHINE_SSL_CERT.cfg
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store MACHINE_SSL_CERT --alias __MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store MACHINE_SSL_CERT --alias __MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert /storage/certmanager/MACHINE_SSL_CERT.crt --key /storage/certmanager/MACHINE_SSL_CERT.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store MACHINE_SSL_CERT --alias __MACHINE_CERT

Generate machine certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/machine.priv --pubkey=/storage/certmanager/machine.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/machine.priv --cert=/storage/certmanager/machine.crt --config=/var/tmp/vmware/machine.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/machine.crt --name machine-6368669f-591a-44fa-bfb3-a76b166bfed6 --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store machine --alias machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store machine --alias machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store machine --alias machine --cert /storage/certmanager/machine.crt --key /storage/certmanager/machine.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store machine --alias machine

Generate web client certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vsphere-webclient.priv --pubkey=/storage/certmanager/vsphere-webclient.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vsphere-webclient.priv --cert=/storage/certmanager/vsphere-webclient.crt --config=/var/tmp/vmware/vsphere-webclient.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vsphere-webclient.crt --name vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6 --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vsphere-webclient --alias vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vsphere-webclient --alias vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert /storage/certmanager/vsphere-webclient.crt --key /storage/certmanager/vsphere-webclient.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vsphere-webclient --alias vsphere-webclient

Generate vpxd certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vpxd.priv --pubkey=/storage/certmanager/vpxd.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vpxd.priv --cert=/storage/certmanager/vpxd.crt --config=/var/tmp/vmware/vpxd.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vpxd.crt --name vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6 --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd --alias vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vpxd --alias vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd --alias vpxd --cert /storage/certmanager/vpxd.crt --key /storage/certmanager/vpxd.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd --alias vpxd

Generate vpxd-extension certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vpxd-extension.priv --pubkey=/storage/certmanager/vpxd-extension.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vpxd-extension.priv --cert=/storage/certmanager/vpxd-extension.crt --config=/var/tmp/vmware/vpxd-extension.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vpxd-extension.crt --name vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6 --login [email protected] --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd-extension --alias vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vpxd-extension --alias vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert /storage/certmanager/vpxd-extension.crt --key /storage/certmanager/vpxd-extension.priv

Update EAM and autodeploy with vpxd-extension certificate

/usr/bin/python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -s is-dhcp36-107.isl.vmware.com -c /storage/certmanager/vpxd-extension.crt -k /storage/certmanager/vpxd-extension.priv - [email protected] -p *****
/usr/bin/python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.rbd -s is-dhcp36-107.isl.vmware.com -c /storage/certmanager/vpxd-extension.crt -k /storage/certmanager/vpxd-extension.priv - [email protected] -p *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd-extension --alias vpxd-extension
service-control --stop --ignore  --all
service-control --start  --all

VMware Converter Workflow for Linux conversions.

Step 1: Validate source

  • connect to source Linux VM via SSH (port 22 by default/specify port number in the IP address field if using custom ports
  • The account used must be a part of sudoers (must be able to run a sudo command without prompting for a password.
add the below line visudo
nik ALL=(ALL) NOPASSWD:ALL

Note: User "nik" to be replaced with the user have you on your setup.

Here’s an example from my setup

nik@mail:~$ sudo cat /etc/sudoers
#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults        env_reset
Defaults        mail_badpass
Defaults        secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification
root    ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d
nik ALL=(ALL) NOPASSWD:ALL

Step 2: Validate Destination

  • connect to ESXi host/vCenter (port 80, 443, 902)
  • Polls for avilable Host (vCenter), Datastore, Networking portgroups, Space.
  • Virtual machine options, Select what Data to copy
  • Hardware resource (like vCPU, Memory, nics
  • Advanced Options like Power off source, Power on destination, Install tools
  • IP details for The Helper VM (if there is no DHCP on the enveronmnet)

A Helper VM is a standby operating environment (live boot) which needs a temporary IP address. The VM helps with the conversion and needs to be able to communicate with the VMware converter server (443) and the source virtual machine (22).

When the job is submitted, Converter creates a dummy virtual machine on the esxi host and boots it via ISO (helper VM iso: converter-helper-vm-x64.iso or converter-helper-vm.iso found at C:\Program Files (x86)\VMware\VMware vCenter Converter Standalone\

By Default, the root/user login for the helper VM is disabled. Inorder to enable this, change the config file located at:
C:\ProgramData\VMware\VMware vCenter Converter Standalone\converter-worker.xml

C:\ProgramData\VMware\VMware vCenter Converter Standalone\converter-worker.xml 

change:    <useSourcePasswordInHelperVm>false</useSourcePasswordInHelperVm>
to 
<useSourcePasswordInHelperVm>true</useSourcePasswordInHelperVm>

Restart all converter related services via services.msc

The root password will now be the same as the password that was used to connect to the source VM

when the Helper VM is on the network, it attempts to ssh into the source linux VM and runs a query like that to the below:

 ssh user@source_linux_IP -p 22 "sudo tar --one-file-system --sparse -C '/' -cf - ." | /us r/bin/tar --numeric-owner --delay-directory-restore -C '/home/p2vtest/' -xf 

Similarly, the other partitions are copied over to the Helper VM (review the helper-VM logs for others)

Once all volume’s are copied over, the filesystem (root and other volume’s, sda, sdb) are re-mapped with the respective path

The bootloader/GRUB is rebuilt (This is native to the version of linux have you, Running custom kernel can break this process)

 /usr/lib/vmware-converter/installGrub.sh