vCenter - File system /storage/log is low on storage space

After a recent VCSA reboot, I was seeing the infamous no healthy upstream error from vCenter.

The first place to check for issues like this is VMware's Virtual Appliance Management Interface (VAMI), located by default via HTTPS on port 5480. An administrator can use the appliance root password for this particular interface.

When reviewing this issue with the VAMI, I saw the following error:

Now, VCSA by design automatically rotates most logs available on the appliance using the open-source tool logrotate, but nothing in this directory appears to be managed:

1root@vcenter# grep /storage/log/etc/logrotate.d/*

I'd say this particular log partition is going to need some manual cleanup every now and then. To open up the CLI, SSH into vCenter and execute the following command:

1Command> shell  
2Shell access is granted to root

First, let's get an idea of how full the disks are:

Note: The -m switch converts units into Megabytes

 1root@vcenter[~]# df -m  
 2Filesystem 1M-blocks Used Available Use% Mounted on  
 3devtmpfs 5982 0 5982 0% /dev  
 4tmpfs 5993 1 5992 1% /dev/shm  
 5tmpfs 5993 2 5992 1% /run  
 6tmpfs 5993 0 5993 0% /sys/fs/cgroup  
 7/dev/sda3 46988 7199 37374 17% /  
 8tmpfs 5993 5 5988 1% /tmp  
 9/dev/mapper/dblog\_vg-dblog 15047 185 14080 2% /storage/dblog  
10/dev/mapper/vtsdb\_vg-vtsdb 10008 68 9412 1% /storage/vtsdb  
11/dev/mapper/vtsdblog\_vg-vtsdblog 4968 36 4661 1% /storage/vtsdblog  
12/dev/sda2 120 30 82 27% /boot  
13/dev/mapper/log\_vg-log 10008 9475 6 100% /storage/log  
14/dev/mapper/core\_vg-core 25063 45 23723 1% /storage/core  
15/dev/mapper/db\_vg-db 10008 507 8974 6% /storage/db  
16/dev/mapper/updatemgr\_vg-updatemgr 100273 1953 93185 3% /storage/updatemgr  
17/dev/mapper/netdump\_vg-netdump 985 3 915 1% /storage/netdump  
18/dev/mapper/lifecycle\_vg-lifecycle 100273 3364 91775 4% /storage/lifecycle  
19/dev/mapper/autodeploy\_vg-autodeploy 10008 37 9444 1% /storage/autodeploy  
20/dev/mapper/imagebuilder\_vg-imagebuilder 10008 37 9444 1% /storage/imagebuilder  
21/dev/mapper/seat\_vg-seat 10008 1185 8295 13% /storage/seat  
22/dev/mapper/archive\_vg-archive 50133 16373 31185 35% /storage/archive

The log partition is definitely full. To take an inventory of disk usage, we'll use the du utility, with the s (summarize) and m (megabytes) switches enabled, and then pass the output to sort with the n (numerical) and r (reverse) switches enabled to focus on the most important first.

1root@vcenter[/]# du -sm /storage/log/vmware/\* | sort -n -r  
22578 /storage/log/vmware/eam  
32286 /storage/log/vmware/lookupsvc  
4785 /storage/log/vmware/sso  
5781 /storage/log/vmware/vsphere-ui  
6530 /storage/log/vmware/vmware-updatemgr

Examining these folders further, quite a few of these are old and never rotated. VMware provides the following guidance on what's safe or isn't. Generally, Linux has issues with files being deleted out from under it, so obviously rotated logs can be safely removed. If this is a production system, I'd recommend calling VMware GSS instead of taking it upon yourself. The above command (du -sm * | sort -nr) can be used in any working directory to see what is filling up the logs the most. Here are a few examples of what I deleted to make room:

1rm -rf /storage/log/vmware/eam/web/localhost-2020-*  
2rm -rf /storage/log/vmware/eam/web/localhost_access.2020*  
3rm -rf /storage/log/vmware/eam/web/catalina-2020*

From here, I like to verify that space is cleared:

1root@vcenter[/]# df -m | grep /storage/log  
2/dev/mapper/log_vg-log 10008 5793 3688 62% /storage/log

Catalina andTomcat are names for the same thing. This software package proxies inbound HTTP requests to specific applications, allowing many developers to build code without having to construct a soup-to-nuts HTTP server. Other similar (but more recent) projects include Python's Flask.

With HTTP Proxies and servers, it is useful to keep comprehensive records indicating "who did what", both for security reasons ("whodunit") and for debugging reasons. As a result, Tomcat is a serious log-hog wherever it exists, and it almost never reviews old logs. This is why I evaluated the change as relatively safe.

If this was not an appliance, I would have added a logrotate spec to automatically delete old files from this directory, but it is not recommended to alter VCSA in this way.