Full MAINSYS File System Due to IPMI SEL Logs

I recently encountered an error on one of my ESXi 4.1 lab hosts trying to make a configuration change to the host. The error image is shown below and the text of the error stack pointed to full root file system:

Call "HostNetworkSystem.UpdateNetworkConfig" for object "networkSystem" on ESXi "esx01" failed.
    Operation failed, diagnostics report: Error interacting with configuration file /etc/vmware/esx.conf: 
    Write failed during Unlock.  This is likely due to a full or read-only filesystem.  
    Original Error was: Error interacting with configuration file /etc/vmware/esx.conf: 
    Unable to write to file /etc/vmware/esx.conf.vVPNWW while saving /etc/vmware/esx.conf operation aborted.  
    It is likely this was caused by a Full Disk.

I used the command vdf -h to confirm the error message and as shown in the below output.

~ # vdf -h
    Tardisk                  Space      Used
    SYS1                      201M      201M
    SYS2                       55M       55M
    SYS3                        1M        1M
    SYS4                       12K       12K
    SYS5                       12K       12K
Ramdisk                   Size      Used Available Use% Mounted on
    MAINSYS                    32M       32M        0B 100% --
    tmp                       192M      112K      191M   0% --
    updatestg                 750M       64K      749M   0% --
    hostdstats                 78M        3M       74M   4% --

I used the find command to determine that the problem files were sel and sel.raw in the folder /var/log/ipmi/0. I deleted the files, but after a few minutes, the files filled the MAINSYS ramdisk again. I found that the file sel was completely made up of the repeating entry System Boot: and sel.raw was made up of ef ef …

These files are generated from System Event Log IPMI data from the host so I accessed the host’s management interface. Oddly I found that the SEL was empty. However on a whim a clicked the Clear Event Log link, deleted the sel and sel.raw files and the files did not reappear. As part of the troubleshooting for this I did reset the host’s ESXi configuration and reinstalled ESXi as well but these actions did not help.

Leave a Comment

Your email address will not be published. Required fields are marked *