login to the VRB appliance console (via SSH).
Execute the following commands
/usr/ITFM-Cloud/va-tools/bin/db-client.sh
update itfm_global_property set property_value = 'INR' where property_name = 'itfm.currency';
Refresh the vRBC UI
login to the VRB appliance console (via SSH).
Execute the following commands
/usr/ITFM-Cloud/va-tools/bin/db-client.sh
update itfm_global_property set property_value = 'INR' where property_name = 'itfm.currency';
Refresh the vRBC UI
on the IAAS run the below
cd " C:\Program Files (x86)\VMware\vCAC\Agents\vCenter"
DynamicOps.Vrm.VRMencrypt.exe VRMAgent.exe.config get
DynamicOps.Vrm.VRMencrypt.exe VRMAgent.exe.config set doDeletes FALSE
output
C:\Program Files (x86)\VMware\vCAC\Agents\vCenter>DynamicOps.Vrm.VRMencrypt.exe VRMAgent.exe.config get
managementEndpointName: vCenter
doDeletes: FALSE
su - postgres
cd /opt/vmware/vpostgres/current/bin
./psql
In psql console run,
\c vcac
Login to VRops, Click on Administration>Licencing (under management) > edit the product licensing group.
Click next on the “select license keys tab”,
in the “add members” tab, scroll to the bottom and look for “objects always to exclude”
Expand “vCenter adapter”
Expand “host system” and select the host that you wish to exclude
Note: if you wish to deselect everything under the cluster (cascade) then Click on the down arrow next to “add” and select “selected objects and all its descendants”
Click next and finish to complete wizard
IAAS.ntitta.in (SQL Server: db.ntitta.in)
(10032) The MS DTC on the IaaS database server is not configured correctly for network access. If the servers that host MSDTC have been cloned their CID might not be unique (located in HKEY_CLASSES_ROOT/CID) thus causing this issue. To regenerate its CID values the MSDTC must be reinstalled. Details: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025) Details: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025)
IAAS.ntitta.in\WEB
(10032) The MS DTC on the IaaS database server is not configured correctly for network access. If the servers that host MSDTC have been cloned their CID might not be unique (located in HKEY_CLASSES_ROOT/CID) thus causing this issue. To regenerate its CID values the MSDTC must be reinstalled. Details: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025) Details: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025)
IAAS.ntitta.in\Manager Service
(10032) The MS DTC on the IaaS database server is not configured correctly for network access. If the servers that host MSDTC have been cloned their CID might not be unique (located in HKEY_CLASSES_ROOT/CID) thus causing this issue. To regenerate its CID values the MSDTC must be reinstalled. Details: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025) Details: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025)
Running through the installation and installation of MSDTC by following the instructions on KB https://kb.vmware.com/s/article/59422 did not help
in order to resolve this, I checked the component services on both IIAS and the DB server
GO to Start and then search for component services
Component services> Computers> My Computer> Distributed Translation Coordinator>
Right click on Local DTC> Select Properties>Click on security
In my instance, the allow inbound connection was disabled on the SQL instance. enable the allow inbound and hit apply, the wizard will automatically restart the DTC and then re-run the wizard.
Note: All the options as seen below must be enabled in order for the workflow to complete successfully
Re-run the validation and it was successfull:
VMware Content library, A unique way to make VM templates/ISO’s available across multiple vCenter’s. However, It does not handle Datacenter segregation very well when storing the contents on NFS/VMFS.
A subscribed library would need dedicated storage space and would be pointless in my setup as I had presented the NFS volume to all host.
I have an NFS volume presented across all the host across several Data center. Although the NFS UUID and the name of the datastore are the same (ISO), on the vCenter database, this is stored with a different datacenter ID as they are segregated by the datacenter object.
id | name | datacenter_id
-----+----------------+---------------
157 | Template | 77
158 | SlowBro_400 | 77
159 | ISO | 77
160 | SharedLUN | 77
161 | 10.154 | 77
156 | template | 77
13 | SlowBro_legasy | 2
14 | 10.128 | 2
91 | ISO | 2
16 | Template | 2
92 | is-tse-d129_1 | 2
12 | SlowBro_400 | 2
VCDB=# select * from vpx_entity where id=77;
id | name | type_id | parent_id
----+--------------+---------+-----------
77 | BLR | 8 | 1
(1 row)
VCDB=# select * from vpx_entity where id=2;
id | name | type_id | parent_id
----+------------+---------+-----------
2 | HYD | 8 | 1
(1 row)
With this configuration, When I had created a content library, as per the db, this was only referencing to once of the site datastore, IE: we only see iso (id 159) from BLR (id 77) datacenter.
What this means is The content library objects can only be deployed to one of the datacenter rather than being able to deploy them across all datacenter’s on the vCenter.
VCDB=# select * from cl_storage;
id | storageuri | type
--------------------------------------+--------------------------------------------------------------+-----------
a0018db8-f630-4c04-b0f1-30c900ad691c | Datastore:datastore-159:481639ff-d88d-4622-8872-ec6856e6b157 | Datastore
bf8e8dcb-5b28-4b03-863e-89308bc8c501 | Datastore:datastore-157:481639ff-d88d-4622-8872-ec6856e6b157 | Datastore
The above table references:
VCDB=# select * from cl_library_storage;
library_id | storage_id
--------------------------------------+--------------------------------------
3336e2ad-8166-4e6a-850d-a9d81c41ba01 | a0018db8-f630-4c04-b0f1-30c900ad691c
946cea65-5cd0-41e0-83ab-17259f690ce1 | bf8e8dcb-5b28-4b03-863e-89308bc8c501
So.. Logically, If I add the other ID for ISO on this table cl_storage and create a referencing record in cl_library_storage then we should be able to use the same content library across all datacenter.
The ID from the above must be unique and must match the two tables. I added the below records (i added another record by incrementing one of the values after the co-relating table.
after change
VCDB=# select * from cl_storage;
id | storageuri | type
--------------------------------------+--------------------------------------------------------------+-----------
a0018db8-f630-4c04-b0f1-30c900ad691c | Datastore:datastore-159:481639ff-d88d-4622-8872-ec6856e6b157 | Datastore
bf8e8dcb-5b28-4b03-863e-89308bc8c501 | Datastore:datastore-157:481639ff-d88d-4622-8872-ec6856e6b157 | Datastore
bf8e8dcb-5b28-4b03-863e-89308bc8c502 | Datastore:datastore-11:481639ff-d88d-4622-8872-ec6856e6b157 | Datastore
VCDB=# select * from cl_library_storage;
library_id | storage_id
--------------------------------------+--------------------------------------
3336e2ad-8166-4e6a-850d-a9d81c41ba01 | a0018db8-f630-4c04-b0f1-30c900ad691c
946cea65-5cd0-41e0-83ab-17259f690ce1 | bf8e8dcb-5b28-4b03-863e-89308bc8c501
946cea65-5cd0-41e0-83ab-17259f690ce1 | bf8e8dcb-5b28-4b03-863e-89308bc8c502
after adding the above records, I am now able to deploy VM’s from the content library across Datacenters.
content library DB schema can be found here:
/usr/lib/vmware-content-library/support/scripts/db/PostgreSQL/cls_unified/cls60.sql
Okay, so the other day I had someone reach out to me for a vCenter rest API issue, Apparently, REST API to any vCenter component using the API explorer or the CLI would return the error
com.vmware.vapi.rest.httpNotFound
Cli: Generate session: (on bash shell): (or simply go grab the complete command from the API explorer)
**edit the below**
VC_ADDRESS=vcenter.domain.local
[email protected]
VC_PASSWORD=password
**do not change the below***
curl -u "$VC_USER:$VC_PASSWORD" \
-X POST \
-k --cookie-jar cookies.txt \
"https://$VC_ADDRESS/rest/com/vmware/cis/session"
**the cookies file should now have a sessionID, use this session in the upcomeing commands***
test VAPI using bash
root@nvcsa-01 [ /tmp ]# curl -X GET --header 'Accept: application/json' --header 'vmware-api-session-id: 71b32ba6c59bc4bc284757b2a0d6e525' 'https://vcsa/rest/vcenter/cluster'
{"name":"com.vmware.vapi.rest.httpNotFound","localizableMessages":[{"defaultMessage":"Not found.","id":"com.vmware.vapi.rest.httpNotFound"}],"majorErrorCode":404}r
at this time clearly, something is wrong. Looking at the URL: https://vcsa_URL/rest/vcenter
Looking at the rhttp configuration:
root@VCSA [ /etc/vmware-rhttpproxy/endpoints.conf.d ]# grep rest *.conf
vapi-endpoint.conf:/rest local 12346 redirect allow
vapi-endpoint.conf:/site/rest local 12346 redirect allow
Looking at the process:
root@vcsa [ /etc/vmware-rhttpproxy/endpoints.conf.d ]# netstat -anop | grep -i listen | grep 12346
tcp6 0 0 ::1:12346 :::* LISTEN 11956/vmware-vapi-e off (0.00/0/0)
tcp6 0 0 127.0.0.1:12346 :::* LISTEN 11956/vmware-vapi-e off (0.00/0/0)
So it is clear that the service responsible for REST is vapi and the service is running.
looking at the logs did not reveal anything out of the ordinary. in fact it reported nothing. aside from the below
2019-11-27T02:45:19.623+08:00 | INFO | state-manager1 | DefaultStateManager | Invoking http-server
2019-11-27T02:45:19.624+08:00 | INFO | state-manager1 | BaseServerBuilder | Creating endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346
2019-11-27T02:45:19.682+08:00 | WARN | state-manager1 | BaseServerBuilder | Failed to bind /0:0:0:0:0:0:0:1:12346 while testing the endpoint validity
java.net.SocketException: Protocol family unavailable
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.Socket.bind(Socket.java:644)
at com.vmware.vapi.endpoint.http.BaseServerBuilder.isPortAccessible(BaseServerBuilder.java:172)
at com.vmware.vapi.endpoint.http.BaseServerBuilder.trimInvalidEndpoints(BaseServerBuilder.java:147)
at com.vmware.vapi.endpoint.http.BaseServerBuilder.populateEndpointSettings(BaseServerBuilder.java:183)
at com.vmware.vapi.endpoint.http.BaseServerBuilder.createServer(BaseServerBuilder.java:233)
at com.vmware.vapi.endpoint.http.BaseServerBuilder.buildInitial(BaseServerBuilder.java:75)
at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)
at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)
at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
okay, so this made clear to me that the application “default” had problems with a port bind, (bind /0:0:0:0:0:0:0:1:12346), so I took a look at the configuration file /etc/vmware-vapi/endpoint.properties, removed the ::1 from the config (I don’t think this is a problem TBH since all the other web apps load up and is accessible via rest)
So I bumped up the VAPI logging (instructions here) and found the below
Looking deeper at the logs with trivia enabled, I found the below when the REST calls failed:
2019-11-27T04:21:10.398+08:00 | DEBUG | vAPI-I/O dispatcher-1 | JsonServerConnection | Sending JSON response of size 50
2019-11-27T04:21:21.213+08:00 | DEBUG | jetty-default-35 | RequestDispatcher | method=GET, uriInfo=/vcenter/datacenter
2019-11-27T04:21:21.214+08:00 | DEBUG | jetty-default-35 | UriLocatorImpl | Matched uriTemplates are not found for requesturi = /vcenter/datacenter
2019-11-27T04:21:21.215+08:00 | DEBUG | jetty-default-35 | RestMainServlet | Failed to process request.
RestException [majorErrorCode=404, messageId=com.vmware.vapi.rest.MessageId@1721d534, params=[], message=null]
at com.vmware.vapi.rest.RequestDispatcher.dispatch(RequestDispatcher.java:64)
at com.vmware.vapi.endpoint.servlet.rest.RestMainServlet.service(RestMainServlet.java:44)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.vmware.vapi.endpoint.common.ProxyServlet.service(ProxyServlet.java:50)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:471)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:323)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:293)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at com.vmware.vapi.endpoint.http.RequestSizeFilter.doFilter(RequestSizeFilter.java:59)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)
Okay, so now we are getting somewhere. “Matched uriTemplates are not found for requesturi = /vcenter/datacenter”
I quickly installed jxplorer and reviewed the endpoints for vApi (configuration>site>%site_name%>lookupService>ServiceRegisteration)
Basically, I was looking for the endpoints registered as serviceType.product=com.vmware.cis serviceType.type=cs.vapi
Basically, you wanna look for an endpoint type: .protocol=vapi.json.http and ID com.vmware.vapi.vcenter under the vCenter’s vapi instance similar to the below which was missing on the problematic vCenter.
note that the endpoint name does not really matter. On the problematic vCenter, there existed endpoint9 but it was for a different endpoint.
So what is the difference?
I quickly grabbed the service ID for vApi service for both the working and nonworking instance as a spec using the below command: (replace the id with the service ID on your instance)
/usr/lib/vmidentity/tools/scripts/lstool.py get --url "https://localhost/lookupservice/sdk" --id "f8ab1f69-c73a-4d94-9ac8-7bf85308954e" --no-check-cert --as-spec > /tmp/spec.txI
I observed the below endpoint where missing on the broken vCenter:
endpoint9.type.id=com.vmware.vapi.vcenter <----------------------------
endpoint9.url=http://localhost:12346/vcenter
endpoint9.ssltrust0=
endpoint9.data0.key=com.vmware.vapi.metadata.authentication.remote
endpoint9.data0.value=http://localhost:12346/vcenter
endpoint9.data1.key=com.vmware.vapi.metadata.metamodel.remote
endpoint9.data1.value=http://localhost:12346/vcenter
endpoint9.data2.key=com.vmware.vapi.metadata.cli.remote
endpoint9.data2.value=http://localhost:12346/vcenter
On the broken vCenter VAPi endpoint spec, I added the above as endpoint12 (11 was in use), Saved the file and re-imported it back in using the below command
/usr/lib/vmidentity/tools/scripts/lstool.py reregister --spec spec.txt --url https://localhost/lookupservice/sdk --user [email protected] --password "Admin!23" --id "f8ab1f69-c73a-4d94-9ac8-7bf85308954e" --no-check-cert
Note that the value for “endpoint9.ssltrust0=” should be filled up with the contents found in the spec file
Restarted the VAPI service and VOLA! Rest API for vCenter started to work!!!!