Memory cgroup out of memory on h3c cas

V600R001B03D059 V7.0 E0720 Enterprise 50587

Symptoms

部分虚拟机经常无故重启。

reboot

Cause

查看虚拟机性能视图，观察到虚拟机故障重启时 CPU 短暂飙升至 100% 。

cpu 1
cpu 2
cpu 3

分析 cvk 节点日志，发现 OOM (Out of Memory) 报错，虚拟机进程被终止。

./vms/logdir/var/log/messages:Mar 29 10:48:38 cvknode5 kernel: Memory cgroup out of memory: Kill process 2734854 (kvm) score 71 or sacrifice child
./vms/logdir/var/log/messages:Mar 29 10:48:39 cvknode5 kernel: Memory cgroup out of memory: Kill process 2907014 (kvm) score 71 or sacrifice child
./vms/logdir/var/log/messages:Mar 29 10:50:45 cvknode5 kernel: Memory cgroup out of memory: Kill process 2022420 (kvm) score 70 or sacrifice child
./vms/logdir/var/log/messages:Mar 29 10:50:46 cvknode5 kernel: Memory cgroup out of memory: Kill process 3520983 (kvm) score 70 or sacrifice child

./vms/logdir/var/log/messages:Mar 29 10:48:38 cvknode5 kernel: memory: usage 460324856kB, limit 460324864kB, failcnt 46718
./vms/logdir/var/log/messages:Mar 29 10:48:38 cvknode5 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
./vms/logdir/var/log/messages:Mar 29 10:48:38 cvknode5 kernel: kmem: usage 1359532kB, limit 9007199254740988kB, failcnt 0
./vms/logdir/var/log/messages:Mar 29 10:48:39 cvknode5 kernel: memory: usage 460324864kB, limit 460324864kB, failcnt 64659
./vms/logdir/var/log/messages:Mar 29 10:48:39 cvknode5 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
./vms/logdir/var/log/messages:Mar 29 10:48:39 cvknode5 kernel: kmem: usage 1359228kB, limit 9007199254740988kB, failcnt 0
./vms/logdir/var/log/messages:Mar 29 10:50:45 cvknode5 kernel: memory: usage 460324864kB, limit 460324864kB, failcnt 71603
./vms/logdir/var/log/messages:Mar 29 10:50:45 cvknode5 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
./vms/logdir/var/log/messages:Mar 29 10:50:45 cvknode5 kernel: kmem: usage 1340480kB, limit 9007199254740988kB, failcnt 0
./vms/logdir/var/log/messages:Mar 29 10:50:46 cvknode5 kernel: memory: usage 460324864kB, limit 460324864kB, failcnt 89905
./vms/logdir/var/log/messages:Mar 29 10:50:46 cvknode5 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
./vms/logdir/var/log/messages:Mar 29 10:50:46 cvknode5 kernel: kmem: usage 1340276kB, limit 9007199254740988kB, failcnt 0

./vms/logdir/command.out:free -g
./vms/logdir/command.out----------------------------------
./vms/logdir/command.out-              total        used        free      shared  buff/cache   available
./vms/logdir/command.out-Mem:            754         474         271           4           7         268
./vms/logdir/command.out-Swap:             0           0           0

从日志上看到 cvk 内存使用被限制在 460324864kB ，并且已使用到 460324856kB ，failcnt 次数持续增加到 89905 。但是 cvk 内存还有 268G 可用，所以推断 cvk 内存使用被限制额度了。

Solution

修改系统参数配置，上调内存超分比。

params

目前还弄不清这个超分比为啥会限制物理机内存额度，只能说坑。