November 2019 – Nick's Container on the cloud ;)

Content library hack(DB)

Posted on November 28, 2019March 5, 2020 by Nithin Titta

VMware Content library, A unique way to make VM templates/ISO’s available across multiple vCenter’s. However, It does not handle Datacenter segregation very well when storing the contents on NFS/VMFS.

A subscribed library would need dedicated storage space and would be pointless in my setup as I had presented the NFS volume to all host.

So what seems to be the problem?

I have an NFS volume presented across all the host across several Data center. Although the NFS UUID and the name of the datastore are the same (ISO), on the vCenter database, this is stored with a different datacenter ID as they are segregated by the datacenter object.


  id  |      name      | datacenter_id
 -----+----------------+---------------
  157 | Template       |            77
  158 | SlowBro_400    |            77
  159 | ISO            |            77
  160 | SharedLUN      |            77
  161 | 10.154         |            77
  156 | template       |            77
   13 | SlowBro_legasy |             2
   14 | 10.128         |             2
   91 | ISO            |             2
   16 | Template       |             2
   92 | is-tse-d129_1  |             2
   12 | SlowBro_400    |             2

VCDB=# select * from vpx_entity where id=77;
  id |     name     | type_id | parent_id
 ----+--------------+---------+-----------
  77 | BLR          |       8 |         1
 (1 row)


VCDB=# select * from vpx_entity where id=2;
  id |    name    | type_id | parent_id
 ----+------------+---------+-----------
   2 | HYD        |       8 |         1
 (1 row)

With this configuration, When I had created a content library, as per the db, this was only referencing to once of the site datastore, IE: we only see iso (id 159) from BLR (id 77) datacenter.

What this means is The content library objects can only be deployed to one of the datacenter rather than being able to deploy them across all datacenter’s on the vCenter.

VCDB=# select * from cl_storage;
                  id                  |                          storageuri                          |   type
--------------------------------------+--------------------------------------------------------------+-----------
 a0018db8-f630-4c04-b0f1-30c900ad691c | Datastore:datastore-159:481639ff-d88d-4622-8872-ec6856e6b157  | Datastore
 bf8e8dcb-5b28-4b03-863e-89308bc8c501 | Datastore:datastore-157:481639ff-d88d-4622-8872-ec6856e6b157 | Datastore


The above table references:
VCDB=# select * from cl_library_storage;
              library_id              |              storage_id
--------------------------------------+--------------------------------------
 3336e2ad-8166-4e6a-850d-a9d81c41ba01 | a0018db8-f630-4c04-b0f1-30c900ad691c
 946cea65-5cd0-41e0-83ab-17259f690ce1 | bf8e8dcb-5b28-4b03-863e-89308bc8c501

So.. Logically, If I add the other ID for ISO on this table cl_storage and create a referencing record in cl_library_storage then we should be able to use the same content library across all datacenter.

The ID from the above must be unique and must match the two tables. I added the below records (i added another record by incrementing one of the values after the co-relating table.

after change
VCDB=# select * from cl_storage;
                  id                  |                          storageuri                          |   type
--------------------------------------+--------------------------------------------------------------+-----------
 a0018db8-f630-4c04-b0f1-30c900ad691c | Datastore:datastore-159:481639ff-d88d-4622-8872-ec6856e6b157  | Datastore
 bf8e8dcb-5b28-4b03-863e-89308bc8c501 | Datastore:datastore-157:481639ff-d88d-4622-8872-ec6856e6b157 | Datastore
 bf8e8dcb-5b28-4b03-863e-89308bc8c502 | Datastore:datastore-11:481639ff-d88d-4622-8872-ec6856e6b157  | Datastore

VCDB=# select * from cl_library_storage;
              library_id              |              storage_id
--------------------------------------+--------------------------------------
 3336e2ad-8166-4e6a-850d-a9d81c41ba01 | a0018db8-f630-4c04-b0f1-30c900ad691c
 946cea65-5cd0-41e0-83ab-17259f690ce1 | bf8e8dcb-5b28-4b03-863e-89308bc8c501
 946cea65-5cd0-41e0-83ab-17259f690ce1 | bf8e8dcb-5b28-4b03-863e-89308bc8c502

after adding the above records, I am now able to deploy VM’s from the content library across Datacenters.

content library DB schema can be found here:

/usr/lib/vmware-content-library/support/scripts/db/PostgreSQL/cls_unified/cls60.sql

vCenter REST API returns 404 com.vmware.vapi.rest.httpNotFound

Posted on November 27, 2019November 28, 2019 by Nithin Titta

Okay, so the other day I had someone reach out to me for a vCenter rest API issue, Apparently, REST API to any vCenter component using the API explorer or the CLI would return the error

com.vmware.vapi.rest.httpNotFound

Cli: Generate session: (on bash shell): (or simply go grab the complete command from the API explorer)

**edit the below**
 VC_ADDRESS=vcenter.domain.local
 [email protected]
 VC_PASSWORD=password

**do not change the below***
curl -u "$VC_USER:$VC_PASSWORD" \
    -X POST \
    -k --cookie-jar cookies.txt \
    "https://$VC_ADDRESS/rest/com/vmware/cis/session"

**the cookies file should now have a sessionID, use this session in the upcomeing commands***

test VAPI using bash

root@nvcsa-01 [ /tmp ]# curl -X GET --header 'Accept: application/json' --header 'vmware-api-session-id: 71b32ba6c59bc4bc284757b2a0d6e525' 'https://vcsa/rest/vcenter/cluster'
{"name":"com.vmware.vapi.rest.httpNotFound","localizableMessages":[{"defaultMessage":"Not found.","id":"com.vmware.vapi.rest.httpNotFound"}],"majorErrorCode":404}r

at this time clearly, something is wrong. Looking at the URL: https://vcsa_URL/rest/vcenter

Looking at the rhttp configuration:

root@VCSA [ /etc/vmware-rhttpproxy/endpoints.conf.d ]# grep rest *.conf
vapi-endpoint.conf:/rest local 12346 redirect allow
vapi-endpoint.conf:/site/rest local 12346 redirect allow

Looking at the process:

root@vcsa [ /etc/vmware-rhttpproxy/endpoints.conf.d ]# netstat -anop | grep -i listen | grep 12346
tcp6       0      0 ::1:12346               :::*                    LISTEN      11956/vmware-vapi-e off (0.00/0/0)
tcp6       0      0 127.0.0.1:12346         :::*                    LISTEN      11956/vmware-vapi-e off (0.00/0/0)

So it is clear that the service responsible for REST is vapi and the service is running.
looking at the logs did not reveal anything out of the ordinary. in fact it reported nothing. aside from the below

2019-11-27T02:45:19.623+08:00 | INFO  | state-manager1            | DefaultStateManager            | Invoking http-server
2019-11-27T02:45:19.624+08:00 | INFO  | state-manager1            | BaseServerBuilder              | Creating endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346
2019-11-27T02:45:19.682+08:00 | WARN  | state-manager1            | BaseServerBuilder              | Failed to bind /0:0:0:0:0:0:0:1:12346 while testing the endpoint validity
java.net.SocketException: Protocol family unavailable
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
        at java.net.Socket.bind(Socket.java:644)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.isPortAccessible(BaseServerBuilder.java:172)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.trimInvalidEndpoints(BaseServerBuilder.java:147)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.populateEndpointSettings(BaseServerBuilder.java:183)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.createServer(BaseServerBuilder.java:233)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.buildInitial(BaseServerBuilder.java:75)
        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

okay, so this made clear to me that the application “default” had problems with a port bind, (bind /0:0:0:0:0:0:0:1:12346), so I took a look at the configuration file /etc/vmware-vapi/endpoint.properties, removed the ::1 from the config (I don’t think this is a problem TBH since all the other web apps load up and is accessible via rest)

So I bumped up the VAPI logging (instructions here) and found the below

Looking deeper at the logs with trivia enabled, I found the below when the REST calls failed:

2019-11-27T04:21:10.398+08:00 | DEBUG | vAPI-I/O dispatcher-1     | JsonServerConnection           | Sending JSON response of size 50
2019-11-27T04:21:21.213+08:00 | DEBUG | jetty-default-35          | RequestDispatcher              | method=GET, uriInfo=/vcenter/datacenter
2019-11-27T04:21:21.214+08:00 | DEBUG | jetty-default-35          | UriLocatorImpl                 | Matched uriTemplates are not found for requesturi = /vcenter/datacenter
2019-11-27T04:21:21.215+08:00 | DEBUG | jetty-default-35          | RestMainServlet                | Failed to process request.
RestException [majorErrorCode=404, messageId=com.vmware.vapi.rest.MessageId@1721d534, params=[], message=null]
        at com.vmware.vapi.rest.RequestDispatcher.dispatch(RequestDispatcher.java:64)
        at com.vmware.vapi.endpoint.servlet.rest.RestMainServlet.service(RestMainServlet.java:44)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at com.vmware.vapi.endpoint.common.ProxyServlet.service(ProxyServlet.java:50)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:471)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:323)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:293)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at com.vmware.vapi.endpoint.http.RequestSizeFilter.doFilter(RequestSizeFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:748)

Okay, so now we are getting somewhere. “Matched uriTemplates are not found for requesturi = /vcenter/datacenter”

I quickly installed jxplorer and reviewed the endpoints for vApi (configuration>site>%site_name%>lookupService>ServiceRegisteration)
Basically, I was looking for the endpoints registered as serviceType.product=com.vmware.cis serviceType.type=cs.vapi

Basically, you wanna look for an endpoint type: .protocol=vapi.json.http and ID com.vmware.vapi.vcenter under the vCenter’s vapi instance similar to the below which was missing on the problematic vCenter.

note that the endpoint name does not really matter. On the problematic vCenter, there existed endpoint9 but it was for a different endpoint.

So what is the difference?

I quickly grabbed the service ID for vApi service for both the working and nonworking instance as a spec using the below command: (replace the id with the service ID on your instance)

/usr/lib/vmidentity/tools/scripts/lstool.py get --url "https://localhost/lookupservice/sdk" --id "f8ab1f69-c73a-4d94-9ac8-7bf85308954e" --no-check-cert --as-spec > /tmp/spec.txI

I observed the below endpoint where missing on the broken vCenter:

endpoint9.type.id=com.vmware.vapi.vcenter  <----------------------------
endpoint9.url=http://localhost:12346/vcenter
endpoint9.ssltrust0=
endpoint9.data0.key=com.vmware.vapi.metadata.authentication.remote
endpoint9.data0.value=http://localhost:12346/vcenter
endpoint9.data1.key=com.vmware.vapi.metadata.metamodel.remote
endpoint9.data1.value=http://localhost:12346/vcenter
endpoint9.data2.key=com.vmware.vapi.metadata.cli.remote
endpoint9.data2.value=http://localhost:12346/vcenter

On the broken vCenter VAPi endpoint spec, I added the above as endpoint12 (11 was in use), Saved the file and re-imported it back in using the below command

/usr/lib/vmidentity/tools/scripts/lstool.py reregister --spec spec.txt --url https://localhost/lookupservice/sdk --user [email protected] --password "Admin!23" --id "f8ab1f69-c73a-4d94-9ac8-7bf85308954e" --no-check-cert

Note that the value for “endpoint9.ssltrust0=” should be filled up with the contents found in the spec file

Restarted the VAPI service and VOLA! Rest API for vCenter started to work!!!!

Bumping up VAPI logging in vcsa for REST API troubleshooting

Posted on November 26, 2019November 28, 2019 by Nithin Titta

cp /etc/vmware-vapi/endpoint-log4j.properties endpoint-log4j.properties.bak

sed -i -e 's/INFO/trivia/g' /etc/vmware-vapi/endpoint-log4j.properties

log files: /var/log/vmware/vapi/endpoint/endpoint.log

looking for a specific record on a mssql DB

Posted on November 16, 2019December 26, 2020 by Nithin Titta

create a stored procedure using the below

CREATE PROCEDURE FindMyData_String
    @DataToFind NVARCHAR(4000),
    @ExactMatch BIT = 0
AS
SET NOCOUNT ON

DECLARE @Temp TABLE(RowId INT IDENTITY(1,1), SchemaName sysname, TableName sysname, ColumnName SysName, DataType VARCHAR(100), DataFound BIT)

    INSERT  INTO @Temp(TableName,SchemaName, ColumnName, DataType)
    SELECT  C.Table_Name,C.TABLE_SCHEMA, C.Column_Name, C.Data_Type
    FROM    Information_Schema.Columns AS C
            INNER Join Information_Schema.Tables AS T
                ON C.Table_Name = T.Table_Name
        AND C.TABLE_SCHEMA = T.TABLE_SCHEMA
    WHERE   Table_Type = 'Base Table'
            And Data_Type In ('ntext','text','nvarchar','nchar','varchar','char')


DECLARE @i INT
DECLARE @MAX INT
DECLARE @TableName sysname
DECLARE @ColumnName sysname
DECLARE @SchemaName sysname
DECLARE @SQL NVARCHAR(4000)
DECLARE @PARAMETERS NVARCHAR(4000)
DECLARE @DataExists BIT
DECLARE @SQLTemplate NVARCHAR(4000)

SELECT  @SQLTemplate = CASE WHEN @ExactMatch = 1
                            THEN 'If Exists(Select *
                                          From   ReplaceTableName
                                          Where  Convert(nVarChar(4000), [ReplaceColumnName])
                                                       = ''' + @DataToFind + '''
                                          )
                                     Set @DataExists = 1
                                 Else
                                     Set @DataExists = 0'
                            ELSE 'If Exists(Select *
                                          From   ReplaceTableName
                                          Where  Convert(nVarChar(4000), [ReplaceColumnName])
                                                       Like ''%' + @DataToFind + '%''
                                          )
                                     Set @DataExists = 1
                                 Else
                                     Set @DataExists = 0'
                            END,
        @PARAMETERS = '@DataExists Bit OUTPUT',
        @i = 1

SELECT @i = 1, @MAX = MAX(RowId)
FROM   @Temp

WHILE @i <= @MAX
    BEGIN
        SELECT  @SQL = REPLACE(REPLACE(@SQLTemplate, 'ReplaceTableName', QUOTENAME(SchemaName) + '.' + QUOTENAME(TableName)), 'ReplaceColumnName', ColumnName)
        FROM    @Temp
        WHERE   RowId = @i


        PRINT @SQL
        EXEC SP_EXECUTESQL @SQL, @PARAMETERS, @DataExists = @DataExists OUTPUT

        IF @DataExists =1
            UPDATE @Temp SET DataFound = 1 WHERE RowId = @i

        SET @i = @i + 1
    END

SELECT  SchemaName,TableName, ColumnName
FROM    @Temp
WHERE   DataFound = 1
GO

now execute the stored procedure with the object that you would like to search the DB

exec FindMyData_string 'FindME', 0

A self hosted blog

Month: November 2019

Content library hack(DB)

So what seems to be the problem?

vCenter REST API returns 404 com.vmware.vapi.rest.httpNotFound

Bumping up VAPI logging in vcsa for REST API troubleshooting

looking for a specific record on a mssql DB