Pagefault out of memory on RHEL 7.6
RHEL 7.6 3.10.0-957.el7.x86_64
Oracle Database 12.2.0.1 RAC
Symptoms
[109920.333071] BUG: unable to handle kernel paging request at 0000000000040158
[109920.333188] Call Trace:
[109920.333194] [<ffffffffad4c27f6>] ? prepare_to_wait+0x56/0x90
[109920.333196] [<ffffffffad634ab4>] mem_cgroup_oom_synchronize+0xe4/0x570
[109920.333202] [<ffffffffad4d676b>] ? wake_up_q+0x5b/0x80
[109920.333204] [<ffffffffad634390>] ? mem_cgroup_charge_common+0xc0/0xc0
[109920.333209] [<ffffffffad5bab04>] pagefault_out_of_memory+0x14/0x90
[109920.333215] [<ffffffffadb5acf2>] mm_fault_error+0x6a/0x157
[109920.333220] [<ffffffffadb6f7a8>] __do_page_fault+0x3c8/0x500
[109920.333222] [<ffffffffadb6f915>] do_page_fault+0x35/0x90
[109920.333227] [<ffffffffadb6b758>] page_fault+0x28/0x30
|
Cause
Maybe a kernel bug?
~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)
~]# uname -r
3.10.0-957.el7.x86_64
~]# rpm -qa | grep kernel
kernel-tools-libs-3.10.0-957.el7.x86_64
kernel-tools-3.10.0-957.el7.x86_64
kernel-3.10.0-957.el7.x86_64
kernel-headers-3.10.0-957.el7.x86_64
|
Solution
- Upgrade the kernel version to 3.10.0-956.99.1.
~]# ll /soft/kernel-*
-rwxr-x--- 1 root root 50708172 Sep 3 16:44 /soft/kernel-3.10.0-957.99.1.el7.x86_64.rpm
-rwxr-x--- 1 root root 8466796 Sep 3 16:44 /soft/kernel-headers-3.10.0-957.99.1.el7.x86_64.rpm
-rwxr-x--- 1 root root 7545872 Sep 3 16:44 /soft/kernel-tools-3.10.0-957.99.1.el7.x86_64.rpm
-rwxr-x--- 1 root root 7449184 Sep 3 16:44 /soft/kernel-tools-libs-3.10.0-957.99.1.el7.x86_64.rpm
~]# rpm -ivh /soft/kernel-3.10.0-957.99.1.el7.x86_64.rpm
warning: /soft/kernel-3.10.0-957.99.1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...
Updating / installing...
1:kernel-3.10.0-957.99.1.el7
~]# rpm -Uvh /soft/kernel-headers-3.10.0-957.99.1.el7.x86_64.rpm
warning: /soft/kernel-headers-3.10.0-957.99.1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...
Updating / installing...
1:kernel-headers-3.10.0-957.99.1.el################################# [ 50%]
Cleaning up / removing...
2:kernel-headers-3.10.0-957.el7
~]# rpm -Uvh /soft/kernel-tools-*
warning: /soft/kernel-tools-3.10.0-957.99.1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...
Updating / installing...
1:kernel-tools-libs-3.10.0-957.99.1################################# [ 25%]
2:kernel-tools-3.10.0-957.99.1.el7
Cleaning up / removing...
3:kernel-tools-3.10.0-957.el7
4:kernel-tools-libs-3.10.0-957.el7
~]# ll /boot
total 165456
-rw-r--r-- 1 root root 152151 Oct 4 2022 config-3.10.0-957.99.1.el7.x86_64
-rw-r--r--. 1 root root 151918 Oct 5 2018 config-3.10.0-957.el7.x86_64
drwx------ 3 root root 16384 Jan 1 1970 efi
drwx------. 2 root root 21 Sep 3 11:25 grub2
-rw-------. 1 root root 57174609 Sep 3 11:22 initramfs-0-rescue-a54fba7698b14a01a56e7ce3f5cf9c66.img
-rw------- 1 root root 49065124 Sep 3 23:24 initramfs-3.10.0-957.99.1.el7.x86_64.img
-rw------- 1 root root 21252723 Sep 3 13:34 initramfs-3.10.0-957.el7.x86_64.img
-rw------- 1 root root 13928489 Sep 3 21:10 initramfs-3.10.0-957.el7.x86_64kdump.img
-rw-r--r-- 1 root root 314531 Oct 4 2022 symvers-3.10.0-957.99.1.el7.x86_64.gz
-rw-r--r--. 1 root root 314036 Oct 5 2018 symvers-3.10.0-957.el7.x86_64.gz
-rw------- 1 root root 3553729 Oct 4 2022 System.map-3.10.0-957.99.1.el7.x86_64
-rw-------. 1 root root 3543471 Oct 5 2018 System.map-3.10.0-957.el7.x86_64
-rwxr-xr-x. 1 root root 6635920 Sep 3 11:22 vmlinuz-0-rescue-a54fba7698b14a01a56e7ce3f5cf9c66
-rwxr-xr-x 1 root root 6658904 Oct 4 2022 vmlinuz-3.10.0-957.99.1.el7.x86_64
-rwxr-xr-x. 1 root root 6635920 Oct 5 2018 vmlinuz-3.10.0-957.el7.x86_64
|
Rollback.
rpm -evh rpm -evh kernel-3.10.0-957.99.1.el7.x86_64
rpm -Uvh --oldpackage /mnt/Packages/kernel-headers-* /mnt/Packages/kernel-tools-*
|
・ Manually Upgrading the RHEL Minor Kernel
- Monitoring the cpu,memory usage.
