Log Disk Exhaustion on VCSA
VCSA 7.0.3 20051473
Symptoms
Log Disk Exhaustion on 10.
Cause
The /storage/log is being used at 96% capacity.
root@vc [ ~ ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 9.3G 0 9.3G 0% /dev
tmpfs 9.4G 1.4M 9.4G 1% /dev/shm
tmpfs 9.4G 1.2M 9.4G 1% /run
tmpfs 9.4G 0 9.4G 0% /sys/fs/cgroup
/dev/mapper/vg_root_0-lv_root_0 47G 12G 34G 26% /
tmpfs 9.4G 4.8M 9.3G 1% /tmp
/dev/mapper/dblog_vg-dblog 15G 233M 14G 2% /storage/dblog
/dev/mapper/netdump_vg-netdump 985M 2.5M 915M 1% /storage/netdump
/dev/mapper/archive_vg-archive 49G 45G 2.4G 95% /storage/archive
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 37M 9.3G 1% /storage/imagebuilder
/dev/mapper/updatemgr_vg-updatemgr 98G 481M 93G 1% /storage/updatemgr
/dev/sda3 488M 30M 423M 7% /boot
/dev/mapper/autodeploy_vg-autodeploy 9.8G 37M 9.3G 1% /storage/autodeploy
/dev/mapper/vtsdblog_vg-vtsdblog 15G 57M 14G 1% /storage/vtsdblog
/dev/mapper/core_vg-core 49G 200M 47G 1% /storage/core
/dev/mapper/log_vg-log 10G 9.5G 0.4G 96% /storage/log
/dev/mapper/vtsdb_vg-vtsdb 1.4T 109M 1.4T 1% /storage/vtsdb
/dev/sda2 10M 2.2M 7.9M 22% /boot/efi
/dev/mapper/db_vg-db 9.8G 1.7G 7.6G 19% /storage/db
/dev/mapper/lifecycle_vg-lifecycle 98G 4.2G 89G 5% /storage/lifecycle
/dev/mapper/seat_vg-seat 1.4T 1.6G 1.4T 1% /storage/seat
tmpfs 1.0M 0 1.0M 0% /var/spool/snmp
|
Solution 1: Expand the file system
- Login VCSM(:5480) and confirm the disk number corresponding to /storage/log, such as Hard Disk 5.
-
Remove all snapshot of vCenter VM.
-
Edit setting of vCenter VM, expand Hard Disk 5 capacity from 10G to 15G.
-
SSH to VCSA, run the /usr/lib/applmgmt/support/scripts/autogrow.sh script
root@vc [ ~ ]# /usr/lib/applmgmt/support/scripts/autogrow.sh
root@vc [ ~ ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 9.3G 0 9.3G 0% /dev
tmpfs 9.4G 1.4M 9.4G 1% /dev/shm
tmpfs 9.4G 1.2M 9.4G 1% /run
tmpfs 9.4G 0 9.4G 0% /sys/fs/cgroup
/dev/mapper/vg_root_0-lv_root_0 47G 12G 34G 26% /
tmpfs 9.4G 4.8M 9.3G 1% /tmp
/dev/mapper/dblog_vg-dblog 15G 233M 14G 2% /storage/dblog
/dev/mapper/netdump_vg-netdump 985M 2.5M 915M 1% /storage/netdump
/dev/mapper/archive_vg-archive 49G 9.2G 38G 20% /storage/archive
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 37M 9.3G 1% /storage/imagebuilder
/dev/mapper/updatemgr_vg-updatemgr 98G 481M 93G 1% /storage/updatemgr
/dev/sda3 488M 30M 423M 7% /boot
/dev/mapper/autodeploy_vg-autodeploy 9.8G 37M 9.3G 1% /storage/autodeploy
/dev/mapper/vtsdblog_vg-vtsdblog 15G 57M 14G 1% /storage/vtsdblog
/dev/mapper/core_vg-core 49G 200M 47G 1% /storage/core
/dev/mapper/log_vg-log 15G 9.5G 4.6G 68% /storage/log
/dev/mapper/vtsdb_vg-vtsdb 1.4T 109M 1.4T 1% /storage/vtsdb
/dev/sda2 10M 2.2M 7.9M 22% /boot/efi
/dev/mapper/db_vg-db 9.8G 1.7G 7.6G 19% /storage/db
/dev/mapper/lifecycle_vg-lifecycle 98G 4.2G 89G 5% /storage/lifecycle
/dev/mapper/seat_vg-seat 1.4T 1.6G 1.4T 1% /storage/seat
tmpfs 1.0M 0 1.0M 0% /var/spool/snmp
|
Solution 2: Cleanup the largest files
-
Take a snapshot of the vCenter VM.
-
Run the following command to identify the largest files:
root@vc [ ~ ]# find /storage/log -type f -print0 | xargs -0 du -h | sort -rh | head -n 10
|
- Truncate the log file using the following command to free up disk space.
e.g.)
> /storage/log/vmware/vmware-updatemgr/updatemgr-vmon.log.stderr
echo -n > /storage/log/vmware/sca/vmon-0.stderr
|
Reference
・ Increasing the disk space for the vCenter Server Appliance in vSphere 6.5, 6.7, 7.0 and 8.0
・ vCenter log disk exhaustion or /storage/log full
・ vCenter Appliance /storage/log partition full due to /storage/log/vmware/sca/vmon.stderr
・ After update to vCenter Server Appliance 6.7 Update 3r, /storage/log fills up sue to growth in content-library-runtime.log.stdout
