version: VMware VirtualCenter 6.5.0 build-8024368
vpxd logs:
019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=Main] Account name: root
2019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxUtil] [LoadMachineInstanceUuid] Local instance UUID: 6f0cbee2-56b1-43d3-b13a-348208627e07
2019-02-08T22:44:53.614Z error vpxd[7FB5996C4800] [Originator@6876 sub=Main] Init failed. Exception: N7Vmacore15SystemExceptionE(No space left on device)
--> [context]zKq7AVECAAAAADBxegAOdnB4ZAAAeF4rbGlidm1hY29yZS5zbwAAEBcbAMppGAC8HisAtJgaAH6mGgG4w6d2cHhkAAEBxqcBCsinAeEoVQG9+1QBap9TAuAFAmxpYmMuc28uNgABdZdT[/context]
2019-02-08T22:44:53.615Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down
2019-02-08T22:44:53.615Z info vpxd[7FB5996C4800] [Originator@6876 sub=SupportMgr] Wrote uptime information
2019-02-08T22:44:53.616Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Alert:false@ bora/vpx/vpxd/util/vdb.cpp:509
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-8024368, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B5E90]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B1804]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x00178BDB]: Vmacore::Service::Alert(char const*, char const*, int)
--> backtrace[03] vpxd[0x00A367FF]
--> backtrace[04] vpxd[0x0054E418]
--> backtrace[05] vpxd[0x0054FC2F]
--> backtrace[06] vpxd[0x00539F6A]
--> backtrace[07] libc.so.6[0x000205E0]
--> backtrace[08] vpxd[0x00539775]
--> [backtrace end]
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] Registry Item DB 5 value is ''
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] Setting VDB delay statements queue size to 11000 transactions for 11 GB RAM dedicated to vpxd.
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=CryptUtil] [static bool Vpx::Common::CryptUtil::UnmungePasswordToBuffer(VpxDecrypter*, const string&, char*, size_t)] invalid decrypter
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from 00000:0
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType]: Database error: ODBC error: (00000) -
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Error getting configuration info from the database
2019-02-08T22:44:53.617Z warning vpxd[7FB5996C4800] [Originator@6876 sub=Main] Database not initialized. Nothing to unlock
2019-02-08T22:44:53.617Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter nowLooking at file system:
root@cmtolpvctrapp24 [ / ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 8.0K 16G 1% /dev/shm
tmpfs 16G 656K 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda3 11G 8.1G 2.1G 80% /
tmpfs 16G 884K 16G 1% /tmp
/dev/sda1 120M 28M 87M 24% /boot
/dev/mapper/log_vg-log 25G 1.9G 22G 9% /storage/log
/dev/mapper/dblog_vg-dblog 25G 173M 24G 1% /storage/dblog
/dev/mapper/db_vg-db 25G 396M 23G 2% /storage/db
/dev/mapper/seat_vg-seat 197G 5.8G 181G 4% /storage/seat
/dev/mapper/netdump_vg-netdump 9.8G 23M 9.2G 1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 25G 45M 24G 1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr 99G 435M 93G 1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder 25G 45M 24G 1% /storage/imagebuilder
/dev/mapper/core_vg-core 99G 188M 94G 1% /storage/core
looking at inode:
root@cmtolpvctrapp24 [ / ]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 4115632 531 4115101 1% /dev
tmpfs 4117336 4 4117332 1% /dev/shm
tmpfs 4117336 690 4116646 1% /run
tmpfs 4117336 16 4117320 1% /sys/fs/cgroup
/dev/sda3 712704 712704 0 100% /
tmpfs 4117336 77 4117259 1% /tmp
/dev/sda1 32768 305 32463 1% /boot
/dev/mapper/log_vg-log 1638400 14332 1624068 1% /storage/log
/dev/mapper/dblog_vg-dblog 1638400 22 1638378 1% /storage/dblog
/dev/mapper/db_vg-db 1638400 2854 1635546 1% /storage/db
/dev/mapper/seat_vg-seat 13107200 4430 13102770 1% /storage/seat
/dev/mapper/netdump_vg-netdump 655360 11 655349 1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 1638400 13 1638387 1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr 6553600 273 6553327 1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder 1638400 14 1638386 1% /storage/imagebuilder
/dev/mapper/core_vg-core 6553600 15 6553585 1% /storage/core
Now we determine what is consuming the most of inode:
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
875 /usr/bin
1417 /opt/vmware/lib/python2.7/test
1584 /usr/lib/vmware-vsphere-ui/server/work/deployer/s/global/39/0/h5ngc.war/resources/libs/angular-i18n
5362 /usr/share/man/man3
607550 /var/spool/mqueue
determine what is consuming the most of inode:
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
875 /usr/bin
1417 /opt/vmware/lib/python2.7/test
1584 /usr/lib/vmware-vsphere-ui/server/work/deployer/s/global/39/0/h5ngc.war/resources/libs/angular-i18n
5362 /usr/share/man/man3
607550 /var/spool/mqueue <----------------------culprit here
To resolve this, we run the below to clear the inode
find /var/spool/mqueue/ -type f -print0 | xargs -0 rm -f
Check for available node to conform if everything is cleared up.
root@cmtolpvctrapp24 [ / ]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 4115632 531 4115101 1% /dev
tmpfs 4117336 4 4117332 1% /dev/shm
tmpfs 4117336 690 4116646 1% /run
tmpfs 4117336 16 4117320 1% /sys/fs/cgroup
/dev/sda3 712704 105154 607550 15% /
tmpfs 4117336 77 4117259 1% /tmp
/dev/sda1 32768 305 32463 1% /boot
/dev/mapper/log_vg-log 1638400 14332 1624068 1% /storage/log
/dev/mapper/dblog_vg-dblog 1638400 22 1638378 1% /storage/dblog
/dev/mapper/db_vg-db 1638400 2854 1635546 1% /storage/db
/dev/mapper/seat_vg-seat 13107200 4430 13102770 1% /storage/seat
/dev/mapper/netdump_vg-netdump 655360 11 655349 1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 1638400 13 1638387 1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr 6553600 273 6553327 1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder 1638400 14 1638386 1% /storage/imagebuilder
/dev/mapper/core_vg-core 6553600 15 6553585 1% /storage/core