Info

Permalink
Function
smp_call_function_many
First occurrence
2020-09-23
Last occurrence
2025-04-15
State
NEW
Components
kernel
Quality
-17

Statistics

Operating system Count
CentOS 7 5,539
Architecture Count
x86_64 5,539
Related packages Count
kernel 5,539
 0:3.10.0-1160.76.1.el7 1,038
 0:3.10.0-1062.12.1.el7 981
 0:3.10.0-1160.el7 648
 0:3.10.0-1127.13.1.el7 472
 0:3.10.0-1062.el7 275
 0:3.10.0-1062.9.1.el7 262
 0:3.10.0-1160.102.1.el7 247
 0:3.10.0-1160.24.1.el7 181
 0:3.10.0-1160.118.1.el7 153
 0:3.10.0-1160.88.1.el7 133
 0:3.10.0-1127.19.1.el7 132
 0:3.10.0-1160.81.1.el7 118
 0:3.10.0-1160.15.2.el7 103
 0:3.10.0-1160.95.1.el7 94
 0:3.10.0-1160.62.1.el7 86
 0:3.10.0-1160.71.1.el7 48
 0:3.10.0-1160.25.1.el7 41
 0:3.10.0-1160.59.1.el7 40
 0:3.10.0-1160.6.1.el7 37
 0:3.10.0-1160.42.2.el7 35
 0:3.10.0-1160.11.1.el7 34
 0:3.10.0-1160.45.1.el7 32
 0:3.10.0-1160.21.1.el7 29
 0:3.10.0-1160.49.1.el7 27
 0:3.10.0-1160.53.1.el7 27
 0:3.10.0-1160.108.1.el7 23
 0:3.10.0-1160.92.1.el7 23
 0:3.10.0-1127.el7 22
 0:3.10.0-1160.119.1.el7 22
 0:3.10.0-1160.31.1.el7 20
 0:3.10.0-1062.18.1.el7 19
 0:3.10.0-1127.10.1.el7 16
 0:3.10.0-1160.114.2.el7 16
 0:3.10.0-1127.18.2.el7 11
 0:3.10.0-1062.4.3.el7 11
 0:3.10.0-1160.105.1.el7 10
 0:3.10.0-1160.80.1.el7 9
 0:3.10.0-1160.90.1.el7 9
 0:3.10.0-1160.66.1.el7 8
 0:3.10.0-1062.1.1.el7 8
 0:3.10.0-1062.4.1.el7 7
 0:3.10.0-1160.99.1.el7 7
 0:3.10.0-1062.1.2.el7 6
 0:3.10.0-1160.41.1.el7 5
 0:3.10.0-1160.2.2.el7 4
 0:3.10.0-1160.83.1.el7 4
 0:3.10.0-1127.8.2.el7 3
 0:3.10.0-1160.36.2.el7 3

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1169735
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbc516c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbc47e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbc47e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbc5edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbc5ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbc5ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffbc5f90e4
Build id: 3.10.0-1160.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffbc5f96a1
Build id: 3.10.0-1160.el7.x86_64
0x121
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [java:2260]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter uio_ivshmem(OE) uio gsch(OE) redirfs(OE) sunrpc ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console net_failover virtio_scsi virtio_blk failover cirrus drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring drm_panel_orientation_quirks virtio floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 2260 Comm: java Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
task: ffff9e9808850000 ti: ffff9e923f7dc000 task.ti: ffff9e923f7dc000
RIP: 0010:[<ffffffffbc516c0a>]  [<ffffffffbc516c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9e923f7dfcb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000bc47e6a0 RCX: ffff9e98dce20478
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9e923f7dfce8 R08: ffff9e9232353c00 R09: ffffffffbc785f99
R10: ffff9e98dcf1f160 R11: ffffd76ada516000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9e923f7dfc60 R15: 00007fcddbb83000
FS:  00007fccedbdf700(0000) GS:ffff9e98dcf00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcddbb83000 CR3: 0000000163be6000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbc47e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbc47e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbc5edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbc5ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbc5ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffbc5f90e4>] unmap_region+0xf4/0x140
[<ffffffffbc5f96a1>] ? __vma_rb_erase+0x121/0x220
Complete report #1140051
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9cb16c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9ca7e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9ca7e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9cbedd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9cbef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9cbef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff9cbf90e4
Build id: 3.10.0-1160.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff9cbf96a1
Build id: 3.10.0-1160.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff9cbfb6d5
Build id: 3.10.0-1160.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff9cbfb915
Build id: 3.10.0-1160.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff9cbfcbb2
Build id: 3.10.0-1160.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff9d193f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [systemd:1]
Modules linked in: tcp_lp rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw vmw_vsock_vmci_transport nf_conntrack vsock ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 ppdev vmw_balloon btusb snd_ens1371 btrtl snd_rawmidi btbcm snd_ac97_codec crc32_pclmul btintel ac97_bus ghash_clmulni_intel bluetooth snd_seq snd_seq_device aesni_intel lrw gf128mul
glue_helper ablk_helper snd_pcm cryptd joydev pcspkr snd_timer rfkill snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common nfit crc32c_intel mptspi libata scsi_transport_spi libnvdimm mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 14 PID: 1 Comm: systemd Kdump: loaded Tainted: G    B        L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9fcbdcfb0000 ti: ffff9fcbdcbbc000 task.ti: ffff9fcbdcbbc000
RIP: 0010:[<ffffffff9cb16c0a>]  [<ffffffff9cb16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9fcbdcbbfcb0  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000800009ca7e6a0 RCX: ffff9fcfa9660808
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9fcbdcbbfce8 R08: ffff9fcbdd3dec00 R09: ffffffff9cd85f99
R10: ffff9fcfa999f160 R11: fffff90114995200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9fcbdcbbfc60 R15: 00007fcf3c745000
FS:  00007fcf3c72e940(0000) GS:ffff9fcfa9980000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcf3c745000 CR3: 000000052aeac000 CR4: 00000000007407e0
PKRU: 55555554
Call Trace:
[<ffffffff9ca7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9ca7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9cbedd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9cbef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9cbef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff9cbf90e4>] unmap_region+0xf4/0x140
[<ffffffff9cbf96a1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff9cbfb6d5>] do_munmap+0x2a5/0x480
[<ffffffff9cbfb915>] vm_munmap+0x65/0xb0
[<ffffffff9cbfcbb2>] SyS_munmap+0x22/0x30
[<ffffffff9d193f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 75 9d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #813425
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff95b1718a
Build id: 3.10.0-1160.76.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff95a7e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff95a7e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff95bee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff95bf016f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff95bf0293
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff95bf9a84
Build id: 3.10.0-1160.76.1.el7.x86_64
0xf4
8
? xfs_iunlock
xfs 0xffffffffc052f3d4
Build id: 3.10.0-1160.76.1.el7.x86_64
0xe4
9
? __vma_rb_erase
vmlinux 0xffffffff95bfa041
Build id: 3.10.0-1160.76.1.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffff95bfc025
Build id: 3.10.0-1160.76.1.el7.x86_64
0x2a5
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [edr_agent:50238]
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [edr_agent:50238]
Modules linked in: nf_conntrack_netlink tcp_diag udp_diag inet_diag ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables ext4 mbcache jbd2 bridge stp llc ip_set binfmt_misc ppdev joydev sg virtio_balloon parport_pc parport i2c_piix4 pcspkr xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_console virtio_scsi virtio_blk failover cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix
libata virtio_pci floppy virtio_ring serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 4 PID: 50238 Comm: edr_agent Kdump: loaded Tainted: G             L ------------   3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9af7edd15280 ti: ffff9af351a18000 task.ti: ffff9af351a18000
RIP: 0010:[<ffffffff95b1718a>]  [<ffffffff95b1718a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9af351a1bcb0  EFLAGS: 00000202
Modules linked in: nf_conntrack_netlink tcp_diag udp_diag inet_diag ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables ext4 mbcache jbd2 bridge stp llc ip_set binfmt_misc ppdev joydev sg virtio_balloon parport_pc parport i2c_piix4 pcspkr xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_console virtio_scsi virtio_blk failover cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix
libata virtio_pci floppy virtio_ring serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 4 PID: 50238 Comm: edr_agent Kdump: loaded Tainted: G             L ------------   3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9af7edd15280 ti: ffff9af351a18000 task.ti: ffff9af351a18000
RIP: 0010:[<ffffffff95b1718a>]  [<ffffffff95b1718a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9af351a1bcb0  EFLAGS: 00000202
RAX: 0000000000000009 RBX: 0008000095a7ddd0 RCX: ffff9afa5d060478
RAX: 0000000000000009 RBX: 0008000095a7ddd0 RCX: ffff9afa5d060478
RDX: 0000000000000009 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9af351a1bce8 R08: ffff9af3ac4d3c00 R09: ffffffff95d869f9
R10: ffff9afa5cf1f160 R11: ffffe2c14081d400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9af351a1bc60 R15: 00007f5e4fd00000
FS:  00007f5e4c06b700(0000) GS:ffff9afa5cf00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5e4fd00000 CR3: 0000000101214000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
RDX: 0000000000000009 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9af351a1bce8 R08: ffff9af3ac4d3c00 R09: ffffffff95d869f9
R10: ffff9afa5cf1f160 R11: ffffe2c14081d400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9af351a1bc60 R15: 00007f5e4fd00000
FS:  00007f5e4c06b700(0000) GS:ffff9afa5cf00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5e4fd00000 CR3: 0000000101214000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff95a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff95a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff95bee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff95bf016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff95bf0293>] tlb_finish_mmu+0x23/0x30
[<ffffffff95bf9a84>] unmap_region+0xf4/0x140
[<ffffffffc052f3d4>] ? xfs_iunlock+0xe4/0x120 [xfs]
[<ffffffff95bfa041>] ? __vma_rb_erase+0x121/0x220
[<ffffffff95bfc025>] do_munmap+0x2a5/0x480
Complete report #589280
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa53150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa527d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa527d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa53ea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa53ebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa53ebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffa53f5544
Build id: 3.10.0-1062.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffa53f5b01
Build id: 3.10.0-1062.el7.x86_64
0x121
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [starrocks_be:116143]
Modules linked in: nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 ppdev vmw_balloon snd_ens1371 sb_edac snd_rawmidi snd_ac97_codec iosf_mbi ac97_bus crc32_pclmul snd_seq snd_seq_device ghash_clmulni_intel snd_pcm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_timer snd joydev sg pcspkr soundcore i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix nfit libnvdimm crct10dif_pclmul crct10dif_common mptspi crc32c_intel libata e1000 scsi_transport_spi mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 116143 Comm: starrocks_be Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff9321787b1070 ti: ffff9320e6ba0000 task.ti: ffff9320e6ba0000
RIP: 0010:[<ffffffffa53150ca>]  [<ffffffffa53150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9320e6ba3cb0  EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00080000a527d1d0 RCX: ffff93219d7e04b8
RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9320e6ba3ce8 R08: ffff931af453cc00 R09: ffffffffa557f629
R10: ffff93219d69f160 R11: ffffd9c29fc7d800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9320e6ba3c60 R15: 00007f2b7bf72000
FS:  00007f2b7a079700(0000) GS:ffff93219d680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2b7bf72000 CR3: 000000077ffd4000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa527d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa527d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa53ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa53ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa53ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffa53f5544>] unmap_region+0xf4/0x140
[<ffffffffa53f5b01>] ? __vma_rb_erase+0x121/0x220
Complete report #146039
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff84716cba
Build id: 3.10.0-1062.18.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8467e608
Build id: 3.10.0-1062.18.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8467e67b
Build id: 3.10.0-1062.18.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff847ebc86
Build id: 3.10.0-1062.18.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff847ed74f
Build id: 3.10.0-1062.18.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff847ed873
Build id: 3.10.0-1062.18.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff847f7084
Build id: 3.10.0-1062.18.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff847f7641
Build id: 3.10.0-1062.18.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff847f9655
Build id: 3.10.0-1062.18.1.el7.x86_64
0x2a5
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [VM Thread:2675]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi nfit kvm_intel libnvdimm kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon sg joydev pcspkr parport_pc
parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul crct10dif_common drm crc32c_intel libata mptspi e1000 scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks
CPU: 4 PID: 2675 Comm: VM Thread Kdump: loaded Tainted: G             L ------------   3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9bf288101070 ti: ffff9bf28963c000 task.ti: ffff9bf28963c000
RIP: 0010:[<ffffffff84716cba>]  [<ffffffff84716cba>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9bf28963fcb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008467e440 RCX: ffff9bf29d620678
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9bf28963fce8 R08: ffff9bebf453dc00 R09: ffffffff849819f9
R10: ffff9bf29d71f160 R11: ffffdf3d82a0c800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9bf28963fc60 R15: 00007ffb40985000
FS:  00007ffb117ff700(0000) GS:ffff9bf29d700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffb40985000 CR3: 00000000360d6000 CR4: 00000000000607e0
Call Trace:
[<ffffffff8467e608>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8467e67b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff847ebc86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff847ed74f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff847ed873>] tlb_finish_mmu+0x23/0x30
[<ffffffff847f7084>] unmap_region+0xf4/0x140
[<ffffffff847f7641>] ? __vma_rb_erase+0x121/0x220
[<ffffffff847f9655>] do_munmap+0x2a5/0x480
Complete report #1041961
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa4b170da
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa4a7e258
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa4a7e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa4bee676
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa4bf013f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa4bf0263
Build id: 3.10.0-1160.59.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffa4bf9a54
Build id: 3.10.0-1160.59.1.el7.x86_64
0xf4
8
? xfs_iunlock
xfs 0xffffffffc05753c4
Build id: 3.10.0-1160.59.1.el7.x86_64
0xe4
9
? __vma_rb_erase
vmlinux 0xffffffffa4bfa011
Build id: 3.10.0-1160.59.1.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffffa4bfc045
Build id: 3.10.0-1160.59.1.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffffa4bfc285
Build id: 3.10.0-1160.59.1.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffffa4bfd582
Build id: 3.10.0-1160.59.1.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffffa5199f92
Build id: 3.10.0-1160.59.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffa5199ed5
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa2
15
smpboot_thread_fn
vmlinux 0xffffffffa4ace852
Build id: 3.10.0-1160.59.1.el7.x86_64
0xe2
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [java:681]
c4 
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink sunrpc dm_mirror dm_region_hash dm_log dm_mod ext4 mbcache jbd2 ppdev sg parport_pc virtio_balloon virtio_rng parport i2c_piix4 pcspkr joydev ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops virtio_net ttm net_failover virtio_console virtio_blk failover drm ata_piix libata virtio_pci virtio_ring serio_raw floppy virtio drm_panel_orientation_quirks fuse
CPU: 8 PID: 681 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff8bef3c144200 ti: ffff8bef3c070000 task.ti: ffff8bef3c070000
RIP: 0010:[<ffffffffa4b170da>]  [<ffffffffa4b170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8bef3c073cb0  EFLAGS: 00000202
RAX: 0000000000000014 RBX: ffffffffa4a7ddd0 RCX: ffff8befbf320518
RDX: 0000000000000014 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff8bef3c073ce8 R08: ffff8be119504800 R09: ffffffffa4d86c09
R10: ffff8befbf01f160 R11: ffffe7f13e718800 R12: 0000000000000286
R13: ffff8bef3c073c60 R14: 00007fb3340ba000 R15: 00007fb3340b9000
FS:  00007fb1fc8e6700(0000) GS:ffff8befbf000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb3340b9000 CR3: 0000000fcec62000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa4a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa4a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa4bee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa4bf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa4bf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffa4bf9a54>] unmap_region+0xf4/0x140
[<ffffffffc05753c4>] ? xfs_iunlock+0xe4/0x120 [xfs]
[<ffffffffa4bfa011>] ? __vma_rb_erase+0x121/0x220
[<ffffffffa4bfc045>] do_munmap+0x2a5/0x480
[<ffffffffa4bfc285>] vm_munmap+0x65/0xb0
[<ffffffffa4bfd582>] SyS_munmap+0x22/0x30
[<ffffffffa5199f92>] system_call_fastpath+0x25/0x2a
[<ffffffffa5199ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffa4ace852>] smpboot_thread_fn+0xe2/0x1a0
Complete report #950124
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff98d1e07a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff98c81f48
Build id: 3.10.0-1160.81.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff98c81fbb
Build id: 3.10.0-1160.81.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff98df9f0e
Build id: 3.10.0-1160.81.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffff98dfba4f
Build id: 3.10.0-1160.81.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff98dfbb83
Build id: 3.10.0-1160.81.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff98e05524
Build id: 3.10.0-1160.81.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff98e05b31
Build id: 3.10.0-1160.81.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff98e07b75
Build id: 3.10.0-1160.81.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff98e07db5
Build id: 3.10.0-1160.81.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff98e090e2
Build id: 3.10.0-1160.81.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff993c539a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [360entclient:125542]
Modules linked in: nfring(OE) xt_NFQUEUE nfnetlink_log ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev sg lrw gf128mul joydev pcspkr glue_helper ablk_helper cryptd vmw_balloon vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sd_mod sysfillrect sysimgblt
fb_sys_fops crc_t10dif ttm crct10dif_generic ahci ata_piix libahci drm crct10dif_pclmul crct10dif_common crc32c_intel libata nfit libnvdimm serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: nfring]
CPU: 3 PID: 125542 Comm: 360entclient Kdump: loaded Tainted: GF          OE  ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff93ce8a3b6300 ti: ffff93c723a54000 task.ti: ffff93c723a54000
RIP: 0010:[<ffffffff98d1e07a>]  [<ffffffff98d1e07a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff93c723a57cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000098c81ab0 RCX: ffff93ce9d620650
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff93c723a57ce8 R08: ffff93c7ec55c000 R09: ffffffff98f9c139
R10: ffff93ce9d6df160 R11: ffffc3ead434ec00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff93c723a57c60 R15: 00007fbcc02fc000
FS:  00007fbcd7fff700(0000) GS:ffff93ce9d6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbcc02fc000 CR3: 00000006f17a8000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff98c81f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff98c81fbb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff98df9f0e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff98dfba4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff98dfbb83>] tlb_finish_mmu+0x23/0x40
[<ffffffff98e05524>] unmap_region+0xf4/0x150
[<ffffffff98e05b31>] ? __vma_rb_erase+0x121/0x230
[<ffffffff98e07b75>] do_munmap+0x2a5/0x480
[<ffffffff98e07db5>] vm_munmap+0x65/0xb0
[<ffffffff98e090e2>] SyS_munmap+0x22/0x30
[<ffffffff993c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 95 99 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 19 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #913116
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff97b16c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff97a7e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff97a7e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff97bedd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff97bef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff97bef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff97bf90e4
Build id: 3.10.0-1160.el7.x86_64
0xf4
8
? __schedule
vmlinux 0xffffffff98186e74
Build id: 3.10.0-1160.el7.x86_64
0x594
9
? __vma_rb_erase
vmlinux 0xffffffff97bf96a1
Build id: 3.10.0-1160.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffff97bfb6d5
Build id: 3.10.0-1160.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffff97bfb915
Build id: 3.10.0-1160.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffff97bfcbb2
Build id: 3.10.0-1160.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffff98193f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [connection:30437]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg i2c_piix4 parport_pc vmw_vmci parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper
sd_mod syscopyarea crc_t10dif sysfillrect sysimgblt crct10dif_generic ahci fb_sys_fops ttm nfit libahci libnvdimm crct10dif_pclmul crct10dif_common drm ata_piix crc32c_intel serio_raw vmxnet3 libata drm_panel_orientation_quirks vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 30437 Comm: connection Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff934e467f8000 ti: ffff934e523e4000 task.ti: ffff934e523e4000
RIP: 0010:[<ffffffff97b16c0a>]  [<ffffffff97b16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff934e523e7cb0  EFLAGS: 00000202
RAX: 0000000000000005 RBX: 0008000097a7e6a0 RCX: ffff936c3f360678
RDX: 0000000000000005 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff934e523e7ce8 R08: ffff934dc5b57400 R09: ffffffff97d85f99
R10: ffff936c3f31f160 R11: ffffde1f8782ee00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff934e523e7c60 R15: 00007fb7fb827000
FS:  00007fb7f8afc700(0000) GS:ffff936c3f300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb7c2362000 CR3: 0000000034842000 CR4: 00000000003607e0
Call Trace:
[<ffffffff97a7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff97a7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff97bedd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff97bef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff97bef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff97bf90e4>] unmap_region+0xf4/0x140
[<ffffffff98186e74>] ? __schedule+0x594/0x860
[<ffffffff97bf96a1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff97bfb6d5>] do_munmap+0x2a5/0x480
[<ffffffff97bfb915>] vm_munmap+0x65/0xb0
[<ffffffff97bfcbb2>] SyS_munmap+0x22/0x30
[<ffffffff98193f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 75 98 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #904439
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb791718a
Build id: 3.10.0-1160.76.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb787e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb787e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb79ee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb79f016f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb79f0293
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffb79f9a84
Build id: 3.10.0-1160.76.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffb79fa041
Build id: 3.10.0-1160.76.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffb79fc025
Build id: 3.10.0-1160.76.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffb79fc265
Build id: 3.10.0-1160.76.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffb79fd562
Build id: 3.10.0-1160.76.1.el7.x86_64
0x22
12
tracesys
vmlinux 0xffffffffb7f9a226
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [sisamddaemon:2429]
Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sisap(POE) sisevt(PE) sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon ppdev vmw_vmci sg joydev i2c_piix4 pcspkr parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm nfit ata_piix libnvdimm crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 14 PID: 2429 Comm: sisamddaemon Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9eb723daa100 ti: ffff9eb6f9514000 task.ti: ffff9eb6f9514000
RIP: 0010:[<ffffffffb791718a>]  [<ffffffffb791718a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9eb6f9517cb0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffffffffb787ddd0 RCX: ffff9eb743aa0808
RDX: 0000000000000002 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff9eb6f9517ce8 R08: ffff9eb2b0992400 R09: ffffffffb7b869f9
R10: ffff9eb743d9f160 R11: fffff0be82399600 R12: 0000000000000286
R13: ffff9eb6f9517c60 R14: 00007efe172ba000 R15: 00007efe172b9000
FS:  00007efd87fff700(0000) GS:ffff9eb743d80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efe172b9000 CR3: 0000000600ec8000 CR4: 00000000000407e0
Call Trace:
[<ffffffffb787e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb787e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb79ee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb79f016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb79f0293>] tlb_finish_mmu+0x23/0x30
[<ffffffffb79f9a84>] unmap_region+0xf4/0x140
[<ffffffffb79fa041>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb79fc025>] do_munmap+0x2a5/0x480
[<ffffffffb79fc265>] vm_munmap+0x65/0xb0
[<ffffffffb79fd562>] SyS_munmap+0x22/0x30
[<ffffffffb7f9a226>] tracesys+0xa6/0xcc
Code: 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 55 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #895426
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9f1150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9f07d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9f07d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9f1ea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9f1ebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9f1ebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff9f1f5544
Build id: 3.10.0-1062.el7.x86_64
0xf4
8
? update_curr
vmlinux 0xffffffff9f0e375c
Build id: 3.10.0-1062.el7.x86_64
0x14c
9
? __vma_rb_erase
vmlinux 0xffffffff9f1f5b01
Build id: 3.10.0-1062.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffff9f1f7b45
Build id: 3.10.0-1062.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffff9f1f7d85
Build id: 3.10.0-1062.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffff9f1f9022
Build id: 3.10.0-1062.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffff9f78bede
Build id: 3.10.0-1062.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffff9f78be21
Build id: 3.10.0-1062.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [Index Provenanc:156577]
Modules linked in: binfmt_misc iptable_filter ext4 ppdev mbcache jbd2 joydev sg virtio_balloon parport_pc parport pcspkr i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_scsi virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix virtio_pci virtio_ring libata floppy serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 9 PID: 156577 Comm: Index Provenanc Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffffa022696241c0 ti: ffffa01f51694000 task.ti: ffffa01f51694000
RIP: 0010:[<ffffffff9f1150ca>]  [<ffffffff9f1150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa01f51697cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000c00009f07d1d0 RCX: ffffa024dce204f0
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffffa01f51697ce8 R08: ffffa01e344ad400 R09: ffffffff9f37f629
R10: ffffa024dd05f160 R11: ffffd78f4b873c00 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa01f51697c60 R15: ffffa01f51697c60
FS:  00007f848696d700(0000) GS:ffffa024dd040000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f84ea7a5000 CR3: 00000004e4944000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9f07d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9f07d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9f1ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9f1ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9f1ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffff9f1f5544>] unmap_region+0xf4/0x140
[<ffffffff9f0e375c>] ? update_curr+0x14c/0x1e0
[<ffffffff9f1f5b01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff9f1f7b45>] do_munmap+0x2a5/0x480
[<ffffffff9f1f7d85>] vm_munmap+0x65/0xb0
[<ffffffff9f1f9022>] SyS_munmap+0x22/0x30
[<ffffffff9f78bede>] system_call_fastpath+0x25/0x2a
[<ffffffff9f78be21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 d4 9f f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #880661
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbc11e07a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbc081f48
Build id: 3.10.0-1160.81.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbc081fbb
Build id: 3.10.0-1160.81.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbc1f9f0e
Build id: 3.10.0-1160.81.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbc1fba4f
Build id: 3.10.0-1160.81.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbc1fbb83
Build id: 3.10.0-1160.81.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffbc205524
Build id: 3.10.0-1160.81.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffbc205b31
Build id: 3.10.0-1160.81.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffbc207b75
Build id: 3.10.0-1160.81.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffbc207db5
Build id: 3.10.0-1160.81.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffbc2090e2
Build id: 3.10.0-1160.81.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffbc7c539a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 21s! [C2 CompilerThre:6994]
Modules linked in: af_packet_diag netlink_diag vsock_diag tcp_diag udp_diag inet_diag unix_diag xt_comment xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci parport_pc parport i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi nfit mptscsih crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libnvdimm mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash
dm_log dm_mod fuse
CPU: 4 PID: 6994 Comm: C2 CompilerThre Tainted: G        W  OEL ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff990261a8d280 ti: ffff9902351e8000 task.ti: ffff9902351e8000
RIP: 0010:[<ffffffffbc11e07a>]  [<ffffffffbc11e07a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9902351ebcb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000bc081ab0 RCX: ffff9909ffc20678
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff9902351ebce8 R08: ffff99032c413000 R09: ffffffffbc39c139
R10: ffff9909ffd1f160 R11: fffff9b841ecc800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9902351ebc60 R15: 00007f041c1c3000
FS:  00007f03e8128700(0000) GS:ffff9909ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f041c1c3000 CR3: 000000003059e000 CR4: 00000000000607e0
Call Trace:
[<ffffffffbc081f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbc081fbb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbc1f9f0e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffbc1fba4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbc1fbb83>] tlb_finish_mmu+0x23/0x40
[<ffffffffbc205524>] unmap_region+0xf4/0x150
[<ffffffffbc205b31>] ? __vma_rb_erase+0x121/0x230
[<ffffffffbc207b75>] do_munmap+0x2a5/0x480
[<ffffffffbc207db5>] vm_munmap+0x65/0xb0
[<ffffffffbc2090e2>] SyS_munmap+0x22/0x30
[<ffffffffbc7c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 d5 bc f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 19 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #873384
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff82f1e23e
Build id: 3.10.0-1160.102.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffff82e81fa8
Build id: 3.10.0-1160.102.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff82e8201b
Build id: 3.10.0-1160.102.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff82ffa07e
Build id: 3.10.0-1160.102.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffff82ffbbff
Build id: 3.10.0-1160.102.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff82ffbd33
Build id: 3.10.0-1160.102.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff830058b4
Build id: 3.10.0-1160.102.1.el7.x86_64
0xf4
8
? __wake_up_common_lock
vmlinux 0xffffffff82ed8f30
Build id: 3.10.0-1160.102.1.el7.x86_64
0x90
9
? __vma_rb_erase
vmlinux 0xffffffff83005ec1
Build id: 3.10.0-1160.102.1.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffff83007f15
Build id: 3.10.0-1160.102.1.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffff83008155
Build id: 3.10.0-1160.102.1.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffff83009482
Build id: 3.10.0-1160.102.1.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffff835c539a
Build id: 3.10.0-1160.102.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [grep:24344]
Modules linked in: vmw_vsock_vmci_transport vsock vfat fat ext4 mbcache jbd2 sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel pcspkr aesni_intel lrw joydev gf128mul glue_helper ppdev vmw_balloon ablk_helper sg cryptd parport_pc parport i2c_piix4 vmw_vmci tpm_crb ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ata_piix sd_mod crc_t10dif libahci crct10dif_generic nfit drm libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 drm_panel_orientation_quirks vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 24344 Comm: grep Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.102.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.18227214.B64.2106252220 06/25/2021
task: ffff92a02471a100 ti: ffff929f02554000 task.ti: ffff929f02554000
RIP: 0010:[<ffffffff82f1e23e>]  [<ffffffff82f1e23e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff929f02557cb0  EFLAGS: 00000202
RAX: 0000000000000005 RBX: 0008000082e81b10 RCX: ffff92a13fd606c8
RDX: 0000000000000005 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff929f02557ce8 R08: ffff929e76353800 R09: ffffffff8319c7b9
R10: ffff92a13fd9f160 R11: fffff5b1103a0000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff929f02557c60 R15: 00007eff571a3000
FS:  00007eff5719c740(0000) GS:ffff92a13fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007eff569d3b30 CR3: 00000003d4a72000 CR4: 00000000001607e0
Call Trace:
[<ffffffff82e81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff82e8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff82ffa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff82ffbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff82ffbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffff830058b4>] unmap_region+0xf4/0x150
[<ffffffff82ed8f30>] ? __wake_up_common_lock+0x90/0xc0
[<ffffffff83005ec1>] ? __vma_rb_erase+0x121/0x230
[<ffffffff83007f15>] do_munmap+0x2a5/0x480
[<ffffffff83008155>] vm_munmap+0x65/0xb0
[<ffffffff83009482>] SyS_munmap+0x22/0x30
[<ffffffff835c539a>] system_call_fastpath+0x25/0x2a
Code: 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 19 b5 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 1d 18 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #862075
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8e11760a
Build id: 3.10.0-1127.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8e07e8f8
Build id: 3.10.0-1127.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8e07e96b
Build id: 3.10.0-1127.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8e1eddb6
Build id: 3.10.0-1127.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8e1ef87f
Build id: 3.10.0-1127.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8e1ef9a3
Build id: 3.10.0-1127.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff8e1f91b4
Build id: 3.10.0-1127.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff8e1f9771
Build id: 3.10.0-1127.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff8e1fb7a5
Build id: 3.10.0-1127.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff8e1fb9e5
Build id: 3.10.0-1127.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff8e1fcc82
Build id: 3.10.0-1127.el7.x86_64
0x22
12
tracesys
vmlinux 0xffffffff8e793166
Build id: 3.10.0-1127.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [C1 CompilerThre:17699]
Modules linked in: ip6table_filter ip6_tables xt_owner tcp_diag inet_diag binfmt_misc iptable_filter vmw_vsock_vmci_transport vsock sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper pcspkr cryptd joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod ahci crc_t10dif ata_piix libahci crct10dif_generic drm nfit libata crct10dif_pclmul crct10dif_common libnvdimm crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: talpa_syscallhook]
CPU: 1 PID: 17699 Comm: C1 CompilerThre Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff990bb6c49070 ti: ffff98fcc6068000 task.ti: ffff98fcc6068000
RIP: 0010:[<ffffffff8e11760a>]  [<ffffffff8e11760a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff98fcc606bcb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008e07e730 RCX: ffff990bbf220600
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff98fcc606bce8 R08: ffff98fd29526800 R09: ffffffff8e385099
R10: ffff990bbf25f160 R11: ffffe965c228f800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff98fcc606bc60 R15: 00007fa6374fb000
FS:  00007fa5febf0700(0000) GS:ffff990bbf240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa6374fb000 CR3: 000000095d474000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8e07e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8e07e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8e1eddb6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8e1ef87f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8e1ef9a3>] tlb_finish_mmu+0x23/0x30
[<ffffffff8e1f91b4>] unmap_region+0xf4/0x140
[<ffffffff8e1f9771>] ? __vma_rb_erase+0x121/0x220
[<ffffffff8e1fb7a5>] do_munmap+0x2a5/0x480
[<ffffffff8e1fb9e5>] vm_munmap+0x65/0xb0
[<ffffffff8e1fcc82>] SyS_munmap+0x22/0x30
[<ffffffff8e793166>] tracesys+0xa6/0xcc
Code: 48 63 35 1e 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 0f d5 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 7a c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #781326
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffadd150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffadc7d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffadc7d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaddea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffaddebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffaddebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffaddf5544
Build id: 3.10.0-1062.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffaddf5b01
Build id: 3.10.0-1062.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffaddf7b45
Build id: 3.10.0-1062.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffaddf7d85
Build id: 3.10.0-1062.el7.x86_64
0x65
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [python:127079]
Modules linked in: xt_nat veth binfmt_misc tcp_lp nls_utf8 isofs rfcomm fuse nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev vmw_balloon crc32_pclmul ghash_clmulni_intel btusb snd_ens1371 btrtl snd_rawmidi btbcm snd_ac97_codec btintel bluetooth ac97_bus aesni_intel snd_seq snd_seq_device lrw gf128mul snd_pcm glue_helper ablk_helper cryptd snd_timer joydev rfkill snd pcspkr sg vmw_vmci soundcore i2c_piix4 parport_pc parport
ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix crct10dif_pclmul crct10dif_common nfit scsi_transport_spi crc32c_intel mptscsih libata mptbase libnvdimm serio_raw e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 127079 Comm: python Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffffa10fa13b0000 ti: ffffa10f81768000 task.ti: ffffa10f81768000
RIP: 0010:[<ffffffffadd150ca>]  [<ffffffffadd150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa10f8176bcb0  EFLAGS: 00000202
RAX: 000000000000000a RBX: 00080000adc7d1d0 RCX: ffffa1113189fc10
RDX: 000000000000000a RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa10f8176bce8 R08: ffffa10f7fdd7000 R09: ffffffffadf7f629
R10: ffffa1113161f160 R11: ffffe82a8c79e600 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa10f8176bc60 R15: 00007f80a2fc0000
FS:  00007f8072ffd700(0000) GS:ffffa11131600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f80a2fc0000 CR3: 000000032a6ba000 CR4: 00000000007607f0
PKRU: 55555554
Call Trace:
[<ffffffffadc7d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffadc7d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaddea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaddebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaddebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffaddf5544>] unmap_region+0xf4/0x140
[<ffffffffaddf5b01>] ? __vma_rb_erase+0x121/0x220
[<ffffffffaddf7b45>] do_munmap+0x2a5/0x480
[<ffffffffaddf7d85>] vm_munmap+0x65/0xb0
Complete report #752469
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbdd150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbdc7d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbdc7d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbddea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbddebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbddebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffbddf5544
Build id: 3.10.0-1062.el7.x86_64
0xf4
8
? do_hook_ops
qax_tq_base 0xffffffffc06c375c
Build id: 3.10.0-1062.el7.x86_64
0xdc
9
? __vma_rb_erase
vmlinux 0xffffffffbddf5b01
Build id: 3.10.0-1062.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffffbddf7b45
Build id: 3.10.0-1062.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffffbddf7d85
Build id: 3.10.0-1062.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffffbddf9022
Build id: 3.10.0-1062.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffffbe38bede
Build id: 3.10.0-1062.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffbe38be21
Build id: 3.10.0-1062.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [java:73637]
Modules linked in: qax_tq_base(OE) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_set ipt_rpfilter xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ip_set vxlan ip6_udp_tunnel udp_tunnel veth xt_recent xt_nat xt_statistic ipt_REJECT nf_reject_ipv4 xt_comment xt_mark iptable_mangle ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi overlay(T) sunrpc ppdev pcspkr virtio_balloon joydev i2c_piix4 sg parport_pc parport ip_tables xfs libcrc32c
sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_scsi virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring floppy serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: qax_tq_base]
CPU: 12 PID: 73637 Comm: java Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff957666a10000 ti: ffff9572d0eb0000 task.ti: ffff9572d0eb0000
RIP: 0010:[<ffffffffbdd150ca>]  [<ffffffffbdd150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9572d0eb3cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000bdc7d1d0 RCX: ffff95777ce20568
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9572d0eb3ce8 R08: ffff9568e95d5800 R09: ffffffffbdf7f629
R10: ffffcfe1fb12d2a0 R11: ffffd1aab0e6c200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9572d0eb3c60 R15: 00007fea254d2000
FS:  00007fe9809fe700(0000) GS:ffff95777d100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fea254d2000 CR3: 0000000be73dc000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbdc7d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbdc7d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbddea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbddebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbddebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffbddf5544>] unmap_region+0xf4/0x140
[<ffffffffc06c375c>] ? do_hook_ops+0xdc/0x110 [qax_tq_base]
[<ffffffffbddf5b01>] ? __vma_rb_erase+0x121/0x220
[<ffffffffbddf7b45>] do_munmap+0x2a5/0x480
[<ffffffffbddf7d85>] vm_munmap+0x65/0xb0
[<ffffffffbddf9022>] SyS_munmap+0x22/0x30
[<ffffffffbe38bede>] system_call_fastpath+0x25/0x2a
[<ffffffffbe38be21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 94 be f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #619236
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff943170da
Build id: 3.10.0-1160.42.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9427e258
Build id: 3.10.0-1160.42.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9427e2cb
Build id: 3.10.0-1160.42.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff943ee676
Build id: 3.10.0-1160.42.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff943f013f
Build id: 3.10.0-1160.42.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff943f0263
Build id: 3.10.0-1160.42.2.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff943f9a54
Build id: 3.10.0-1160.42.2.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff943fa011
Build id: 3.10.0-1160.42.2.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff943fc045
Build id: 3.10.0-1160.42.2.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff943fc285
Build id: 3.10.0-1160.42.2.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff943fd582
Build id: 3.10.0-1160.42.2.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff94995f92
Build id: 3.10.0-1160.42.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [dockerd:1872]
Modules linked in: binfmt_misc ip_vs_rr xt_ipvs ip_vs xt_REDIRECT nf_nat_redirect vxlan ip6_udp_tunnel udp_tunnel xt_mark xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper
cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 CyProtectDrv(POE) CyProtectDrvOpen(OE) ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt fb_sys_fops crc_t10dif ahci ttm crct10dif_generic ata_piix libahci drm crct10dif_pclmul nfit crct10dif_common crc32c_intel libata libnvdimm vmxnet3 serio_raw vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 11 PID: 1872 Comm: dockerd Kdump: loaded Tainted: P        W  OEL ------------ T 3.10.0-1160.42.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9990aa5b4200 ti: ffff999070edc000 task.ti: ffff999070edc000
RIP: 0010:[<ffffffff943170da>]  [<ffffffff943170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff999070edfcb0  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 000800009427ddd0 RCX: ffff9990ad7a0790
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff999070edfce8 R08: ffff998df5c6d800 R09: ffffffff94586bd9
R10: ffff9990ad8df160 R11: ffffe55dcb8c4400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff999070edfc60 R15: ffffffff943bd91e
FS:  00007fb7377fe700(0000) GS:ffff9990ad8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb74804d008 CR3: 0000000409904000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9427e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9427e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff943ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff943f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff943f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff943f9a54>] unmap_region+0xf4/0x140
[<ffffffff943fa011>] ? __vma_rb_erase+0x121/0x220
[<ffffffff943fc045>] do_munmap+0x2a5/0x480
[<ffffffff943fc285>] vm_munmap+0x65/0xb0
[<ffffffff943fd582>] SyS_munmap+0x22/0x30
[<ffffffff94995f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 f5 94 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #534602
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff82116e4a
Build id: 3.10.0-1160.15.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8207e868
Build id: 3.10.0-1160.15.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8207e8db
Build id: 3.10.0-1160.15.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff821ede86
Build id: 3.10.0-1160.15.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff821ef94f
Build id: 3.10.0-1160.15.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff821efa73
Build id: 3.10.0-1160.15.2.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff821f9264
Build id: 3.10.0-1160.15.2.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff821f9821
Build id: 3.10.0-1160.15.2.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff821fb855
Build id: 3.10.0-1160.15.2.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff821fba95
Build id: 3.10.0-1160.15.2.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff821fcd92
Build id: 3.10.0-1160.15.2.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff82794f92
Build id: 3.10.0-1160.15.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#103 stuck for 23s! [evdefer/10:14358]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14203(E) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache uinput devlink binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter falcon_lsm_pinned_14108(E) snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm qxl snd_timer ttm drm_kms_helper iTCO_wdt syscopyarea
crc32_pclmul ghash_clmulni_intel aesni_intel lrw sysfillrect gf128mul sysimgblt glue_helper ppdev fb_sys_fops ablk_helper snd iTCO_vendor_support joydev cryptd parport_pc drm virtio_rng pcspkr i2c_i801 lpc_ich soundcore virtio_balloon parport drm_panel_orientation_quirks nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c virtio_net net_failover virtio_console virtio_blk failover crct10dif_pclmul ahci virtio_pci libahci crct10dif_common serio_raw crc32c_intel libata virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
CPU: 103 PID: 14358 Comm: evdefer/10 Kdump: loaded Tainted: P        W   EL ------------   3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.13.0-2.module+el8.4.0+534+4680a14e 04/01/2014
task: ffff8b1f00b84200 ti: ffff8b13a6b74000 task.ti: ffff8b13a6b74000
RIP: 0010:[<ffffffff82116e4a>]  [<ffffffff82116e4a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b13a6b77cb0  EFLAGS: 00000202
RAX: 0000000000000035 RBX: 000800008207e6a0 RCX: ffff8cfbac361428
RDX: 0000000000000035 RSI: 00000000000000ff RDI: 0000000000000000
RBP: ffff8b13a6b77ce8 R08: ffff8b10ac352400 R09: ffffffff82386479
R10: ffff8cfbacfdf160 R11: fffff7083281ea00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8b13a6b77c60 R15: 00007fcdfe03b000
FS:  00007fcdef2a6700(0000) GS:ffff8cfbacfc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcdfe03b000 CR3: 0000000470f76000 CR4: 0000000000760ee0
PKRU: 55555554
Call Trace:
[<ffffffff8207e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8207e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff821ede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff821ef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff821efa73>] tlb_finish_mmu+0x23/0x30
[<ffffffff821f9264>] unmap_region+0xf4/0x140
[<ffffffff821f9821>] ? __vma_rb_erase+0x121/0x220
[<ffffffff821fb855>] do_munmap+0x2a5/0x480
[<ffffffff821fba95>] vm_munmap+0x65/0xb0
[<ffffffff821fcd92>] SyS_munmap+0x22/0x30
[<ffffffff82794f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 de 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 d5 82 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #520961
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffadd170da
Build id: 3.10.0-1160.41.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffadc7e258
Build id: 3.10.0-1160.41.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffadc7e2cb
Build id: 3.10.0-1160.41.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaddee676
Build id: 3.10.0-1160.41.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffaddf013f
Build id: 3.10.0-1160.41.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffaddf0263
Build id: 3.10.0-1160.41.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffaddf9a54
Build id: 3.10.0-1160.41.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffaddfa011
Build id: 3.10.0-1160.41.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffaddfc045
Build id: 3.10.0-1160.41.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffaddfc285
Build id: 3.10.0-1160.41.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffaddfd582
Build id: 3.10.0-1160.41.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffae395f92
Build id: 3.10.0-1160.41.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffae395ed5
Build id: 3.10.0-1160.41.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [f2b/f.sshd:1828]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink nfsv3 nfs fscache joydev i2c_piix4 sg virtio_rng virtio_balloon pcspkr edac_mce_amd openafs(POE) nfsd auth_rpcgss nfs_acl lockd grace binfmt_misc sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net net_failover virtio_console virtio_scsi failover qxl drm_kms_helper crct10dif_pclmul syscopyarea crct10dif_common sysfillrect crc32_pclmul sysimgblt fb_sys_fops crc32c_intel ttm ghash_clmulni_intel ata_piix
drm libata aesni_intel serio_raw glue_helper lrw gf128mul virtio_pci floppy ablk_helper cryptd virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1828 Comm: f2b/f.sshd Kdump: loaded Tainted: P           OE  ------------   3.10.0-1160.41.1.el7.x86_64 #1
Hardware name: oVirt RHEL, BIOS 1.16.0-1.module_el8.7.0+1140+ff0772f9 04/01/2014
task: ffff9052e2f1c200 ti: ffff9052e1cc4000 task.ti: ffff9052e1cc4000
RIP: 0010:[<ffffffffadd170da>]  [<ffffffffadd170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9052e1cc7cb0  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00080000adc7ddd0 RCX: ffff9052f32e0600
RDX: 0000000000000003 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff9052e1cc7ce8 R08: ffff9051382fe800 R09: ffffffffadf86bd9
R10: ffff9052f325f160 R11: fffff5baccc7b200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9052e1cc7c60 R15: 00007f79f88f7000
FS:  00007f79e9be3700(0000) GS:ffff9052f3240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f79f88f7000 CR3: 0000000035ac0000 CR4: 00000000000407e0
Call Trace:
[<ffffffffadc7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffadc7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaddee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaddf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaddf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffaddf9a54>] unmap_region+0xf4/0x140
[<ffffffffaddfa011>] ? __vma_rb_erase+0x121/0x220
[<ffffffffaddfc045>] do_munmap+0x2a5/0x480
[<ffffffffaddfc285>] vm_munmap+0x65/0xb0
[<ffffffffaddfd582>] SyS_munmap+0x22/0x30
[<ffffffffae395f92>] system_call_fastpath+0x25/0x2a
[<ffffffffae395ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 95 ae f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #513188
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa1916c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa187e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa187e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa19edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa19ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa19ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffa19f90e4
Build id: 3.10.0-1160.el7.x86_64
0xf4
8
? do_sync_write
vmlinux 0xffffffffa1a4d060
Build id: 3.10.0-1160.el7.x86_64
0x90
9
? __vma_rb_erase
vmlinux 0xffffffffa19f96a1
Build id: 3.10.0-1160.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffffa19fb6d5
Build id: 3.10.0-1160.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffffa19fb915
Build id: 3.10.0-1160.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffffa19fcbb2
Build id: 3.10.0-1160.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffffa1f93f92
Build id: 3.10.0-1160.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffa1f93ed5
Build id: 3.10.0-1160.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 50s! [DrCOMX:10033]
Modules linked in: tun devlink bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul snd_ens1371 snd_rawmidi ghash_clmulni_intel snd_ac97_codec ppdev ac97_bus snd_seq snd_seq_device vmw_balloon snd_pcm aesni_intel lrw gf128mul snd_timer glue_helper ablk_helper snd cryptd vmw_vmci sg pcspkr joydev soundcore i2c_piix4 parport_pc parport xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm e1000 mptspi scsi_transport_spi mptscsih drm mptbase nfit libnvdimm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 0 PID: 10033 Comm: DrCOMX Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff97f393bb2100 ti: ffff97f2fb258000 task.ti: ffff97f2fb258000
RIP: 0010:[<ffffffffa1916c0a>]  [<ffffffffa1916c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff97f2fb25bcb0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffffffffa187e6a0 RCX: ffff97f45ee9fc10
RDX: 0000000000000002 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff97f2fb25bce8 R08: ffff97edbfd6f400 R09: ffffffffa1b85f99
R10: ffff97f45ee1f160 R11: fffff0d040d76c00 R12: 0000000000000286
R13: ffff97f2fb25bc60 R14: 00007f624a9a8000 R15: 00007f624a9a7000
FS:  00007f623f7fe700(0000) GS:ffff97f45ee00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f624a9a7000 CR3: 00000006c8462000 CR4: 00000000000407f0
Call Trace:
[<ffffffffa187e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa187e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa19edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa19ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa19ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffa19f90e4>] unmap_region+0xf4/0x140
[<ffffffffa1a4d060>] ? do_sync_write+0x90/0xe0
[<ffffffffa19f96a1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffa19fb6d5>] do_munmap+0x2a5/0x480
[<ffffffffa19fb915>] vm_munmap+0x65/0xb0
[<ffffffffa19fcbb2>] SyS_munmap+0x22/0x30
[<ffffffffa1f93f92>] system_call_fastpath+0x25/0x2a
[<ffffffffa1f93ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 55 a2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #475178
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9fd163d2
Build id: 3.10.0-1062.12.1.el7.x86_64
0x102
2
flush_tlb_mm_range
vmlinux 0xffffffff9fc7e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
3
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9fdec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
4
arch_tlb_finish_mmu
vmlinux 0xffffffff9fdedb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
5
tlb_finish_mmu
vmlinux 0xffffffff9fdedc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
6
unmap_region
vmlinux 0xffffffff9fdf7474
Build id: 3.10.0-1062.12.1.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff9fdf7a31
Build id: 3.10.0-1062.12.1.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff9fdf9a75
Build id: 3.10.0-1062.12.1.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffff9fdf9cb5
Build id: 3.10.0-1062.12.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff9fdfaf52
Build id: 3.10.0-1062.12.1.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffffa038dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
12
? system_call_after_swapgs
vmlinux 0xffffffffa038de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [python:119291]
Modules linked in: binfmt_misc xt_multiport nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev virtio_balloon pcspkr sg joydev parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk
virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 119291 Comm: python Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9db349eb9070 ti: ffff9db03a400000 task.ti: ffff9db03a400000
RIP: 0010:[<ffffffff9fd163d2>]  [<ffffffff9fd163d2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9db03a403c20  EFLAGS: 00000202
RAX: 00000000000000a0 RBX: ffff9db03a403bf0 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 00000000000000a0 RDI: 0000000000000282
RBP: ffff9db03a403c70 R08: ffffffffa0413cc8 R09: ffffffff9fe00718
R10: ffff9db3ace1f0a0 R11: fffff6975071ce80 R12: ffffffff9fdc8144
R13: ffff9db03a403cb8 R14: 0000000000000002 R15: 0000000000000000
FS:  00007fef49541740(0000) GS:ffff9db3ace00000(0000) knlGS:0000000000000000
[<ffffffff9fc7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9fdec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9fdedb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9fdedc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff9fdf7474>] unmap_region+0xf4/0x140
[<ffffffff9fdf7a31>] ? __vma_rb_erase+0x121/0x220
[<ffffffff9fdf9a75>] do_munmap+0x2a5/0x480
[<ffffffff9fdf9cb5>] vm_munmap+0x65/0xb0
[<ffffffff9fdfaf52>] SyS_munmap+0x22/0x30
[<ffffffffa038dede>] system_call_fastpath+0x25/0x2a
[<ffffffffa038de21>] ? system_call_after_swapgs+0xae/0x146
Code: 89 de 48 03 14 c5 20 e8 94 a0 48 89 df e8 37 23 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #439203
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9f91720a
Build id: 3.10.0-1160.66.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9f87e258
Build id: 3.10.0-1160.66.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9f87e2cb
Build id: 3.10.0-1160.66.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9f9ee676
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9f9f013f
Build id: 3.10.0-1160.66.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9f9f0263
Build id: 3.10.0-1160.66.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff9f9f9a54
Build id: 3.10.0-1160.66.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff9f9fa011
Build id: 3.10.0-1160.66.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff9f9fc045
Build id: 3.10.0-1160.66.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff9f9fc285
Build id: 3.10.0-1160.66.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff9f9fd582
Build id: 3.10.0-1160.66.1.el7.x86_64
0x22
12
tracesys
vmlinux 0xffffffff9ff9a226
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [metricbeat:5491]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13501(E) snd_hda_codec_generic snd_hda_intel snd_hda_codec iosf_mbi snd_hda_core snd_hwdep snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device aesni_intel lrw snd_pcm gf128mul glue_helper pcspkr snd_timer ppdev ablk_helper snd joydev cryptd virtio_rng soundcore parport_pc i2c_piix4 parport virtio_balloon ip_tables xfs libcrc32c ata_generic pata_acpi virtio_console virtio_blk qxl drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm e1000 libata crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci serio_raw virtio_ring floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 5491 Comm: metricbeat Tainted: P            E  ------------   3.10.0-1160.66.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module+el8.3.0+7860+a7792d29 04/01/2014
task: ffff8b2310694200 ti: ffff8b21a1e20000 task.ti: ffff8b21a1e20000
RIP: 0010:[<ffffffff9f91720a>]  [<ffffffff9f91720a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b21a1e23cb0  EFLAGS: 00000202
RAX: 0000000000000005 RBX: 000800009f87ddd0 RCX: ffff8b233fd5fc10
RDX: 0000000000000005 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff8b21a1e23ce8 R08: ffff8b22ffd4f800 R09: ffffffff9fb869e9
R10: ffff8b233fc1f160 R11: ffffd1dec092a000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8b21a1e23c60 R15: 00007fc009e85000
FS:  00007fbfbbfff700(0000) GS:ffff8b233fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc009e85000 CR3: 00000001b8ef6000 CR4: 00000000001606f0
Call Trace:
[<ffffffff9f87e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9f87e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9f9ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9f9f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9f9f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff9f9f9a54>] unmap_region+0xf4/0x140
[<ffffffff9f9fa011>] ? __vma_rb_erase+0x121/0x220
[<ffffffff9f9fc045>] do_munmap+0x2a5/0x480
[<ffffffff9f9fc285>] vm_munmap+0x65/0xb0
[<ffffffff9f9fd582>] SyS_munmap+0x22/0x30
[<ffffffff9ff9a226>] tracesys+0xa6/0xcc
Code: 48 63 35 9e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 55 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #406442
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9e7170da
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9e67e258
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9e67e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9e7ee676
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9e7f013f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9e7f0263
Build id: 3.10.0-1160.59.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff9e7f9a54
Build id: 3.10.0-1160.59.1.el7.x86_64
0xf4
8
? xfs_iunlock
xfs 0xffffffffc05133c4
Build id: 3.10.0-1160.59.1.el7.x86_64
0xe4
9
? __vma_rb_erase
vmlinux 0xffffffff9e7fa011
Build id: 3.10.0-1160.59.1.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffff9e7fc045
Build id: 3.10.0-1160.59.1.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffff9e7fc285
Build id: 3.10.0-1160.59.1.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffff9e7fd582
Build id: 3.10.0-1160.59.1.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffff9ed99f92
Build id: 3.10.0-1160.59.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [hz._hzInstance_:57026]
Modules linked in: synosnap(OE) veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 btusb snd_rawmidi btrtl snd_ac97_codec btbcm sb_edac ac97_bus btintel snd_seq iosf_mbi
snd_seq_device snd_pcm sg bluetooth crc32_pclmul ghash_clmulni_intel aesni_intel rfkill lrw gf128mul glue_helper ppdev ablk_helper snd_timer snd vmw_balloon joydev pcspkr cryptd soundcore parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix nfit scsi_transport_spi drm mptscsih libata libnvdimm e1000 serio_raw mptbase drm_panel_orientation_quirks fuse
CPU: 10 PID: 57026 Comm: hz._hzInstance_ Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffffa12719ba9080 ti: ffffa127433e4000 task.ti: ffffa127433e4000
RIP: 0010:[<ffffffff9e7170da>]  [<ffffffff9e7170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa127433e7cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009e67ddd0 RCX: ffffa1277d020768
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa127433e7ce8 R08: ffffa122afd88800 R09: ffffffff9e986c09
R10: ffffa1277d29f160 R11: ffffe71c42c9c400 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa127433e7c60 R15: 00007fa35c065000
FS:  00007fa33f7f7700(0000) GS:ffffa1277d280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa35c065000 CR3: 000000063c186000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9e67e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9e67e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9e7ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9e7f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9e7f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff9e7f9a54>] unmap_region+0xf4/0x140
[<ffffffffc05133c4>] ? xfs_iunlock+0xe4/0x120 [xfs]
[<ffffffff9e7fa011>] ? __vma_rb_erase+0x121/0x220
[<ffffffff9e7fc045>] do_munmap+0x2a5/0x480
[<ffffffff9e7fc285>] vm_munmap+0x65/0xb0
[<ffffffff9e7fd582>] SyS_munmap+0x22/0x30
[<ffffffff9ed99f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 35 9f f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #370515
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb7f1519e
Build id: 3.10.0-1062.4.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffb7e7d478
Build id: 3.10.0-1062.4.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb7e7d4eb
Build id: 3.10.0-1062.4.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb7fea756
Build id: 3.10.0-1062.4.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb7fec21f
Build id: 3.10.0-1062.4.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb7fec343
Build id: 3.10.0-1062.4.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffb7ff5b34
Build id: 3.10.0-1062.4.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffb7ff60f1
Build id: 3.10.0-1062.4.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffb7ff8135
Build id: 3.10.0-1062.4.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffb7ff8375
Build id: 3.10.0-1062.4.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffb7ff9612
Build id: 3.10.0-1062.4.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffb858bede
Build id: 3.10.0-1062.4.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [evdefer/3:14855]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13109(E) falcon_lsm_pinned_13005(E) nls_utf8 isofs fuse rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache devlink binfmt_misc falcon_lsm_pinned_12905(E) ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_hda_codec_generic
lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel joydev snd_hda_codec virtio_balloon snd_hda_core sg snd_hwdep snd_seq parport_pc snd_seq_device parport snd_pcm snd_timer snd pcspkr i2c_piix4 soundcore auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw crct10dif_pclmul crct10dif_common virtio_pci crc32c_intel floppy virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal]
CPU: 3 PID: 14855 Comm: evdefer/3 Kdump: loaded Tainted: P            EL ------------   3.10.0-1062.4.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffffa0d716e28000 ti: ffffa0d711364000 task.ti: ffffa0d711364000
RIP: 0010:[<ffffffffb7f1519e>]  [<ffffffffb7f1519e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffffa0d711367cb0  EFLAGS: 00000202
RAX: 0000000000000005 RBX: 00080000b7e7d2b0 RCX: ffffa0d71f160370
RDX: 0000000000000005 RSI: 0000000000000018 RDI: 0000000000000000
RBP: ffffa0d711367ce8 R08: ffffa0a8f6c07c00 R09: ffffffffb817fe99
R10: ffffa0d71f0df160 R11: fffffbf03cb5b400 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa0d711367c60 R15: ffffffffb7e7d2b0
FS:  00007fb0f7ab4700(0000) GS:ffffa0d71f0c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000095fe9003 CR3: 0000002e66b96000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb7e7d478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb7e7d4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb7fea756>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb7fec21f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb7fec343>] tlb_finish_mmu+0x23/0x30
[<ffffffffb7ff5b34>] unmap_region+0xf4/0x140
[<ffffffffb7ff60f1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb7ff8135>] do_munmap+0x2a5/0x480
[<ffffffffb7ff8375>] vm_munmap+0x65/0xb0
[<ffffffffb7ff9612>] SyS_munmap+0x22/0x30
[<ffffffffb858bede>] system_call_fastpath+0x25/0x2a
Code: 66 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d6 b4 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 3d 66 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #364195
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff87316a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8727e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8727e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff873ec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff873edb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff873edc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff873f7474
Build id: 3.10.0-1062.12.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff873f7a31
Build id: 3.10.0-1062.12.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff873f9a75
Build id: 3.10.0-1062.12.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff873f9cb5
Build id: 3.10.0-1062.12.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff873faf52
Build id: 3.10.0-1062.12.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff8798dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff8798de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [360entclient:3552]
Modules linked in: tcp_lp fuse kbase(OE) xt_NFQUEUE nfnetlink_log nfnetlink nfring(OE) vmsecmod(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev sg parport_pc pcspkr parport virtio_balloon joydev i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring serio_raw floppy virtio
drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmsec_nfq]
CPU: 9 PID: 3552 Comm: 360entclient Kdump: loaded Tainted: GF          OEL ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9eaebee5a0e0 ti: ffff9eaeadb50000 task.ti: ffff9eaeadb50000
RIP: 0010:[<ffffffff87316a7a>]  [<ffffffff87316a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9eaeadb53cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008727e2b0 RCX: ffff9eaf7ce20540
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9eaeadb53ce8 R08: ffff9ea0e95dcc00 R09: ffffffff87581839
R10: ffff9eaf7d05f160 R11: ffffdd7dffdada00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9eaeadb53c60 R15: 00007f20dadc8000
FS:  00007f21377e4700(0000) GS:ffff9eaf7d040000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f20dadc8000 CR3: 0000000ff1af8000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8727e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8727e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff873ec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff873edb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff873edc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff873f7474>] unmap_region+0xf4/0x140
[<ffffffff873f7a31>] ? __vma_rb_erase+0x121/0x220
[<ffffffff873f9a75>] do_munmap+0x2a5/0x480
[<ffffffff873f9cb5>] vm_munmap+0x65/0xb0
[<ffffffff873faf52>] SyS_munmap+0x22/0x30
[<ffffffff8798dede>] system_call_fastpath+0x25/0x2a
[<ffffffff8798de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 f4 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #343782
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffadd16a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffadc7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffadc7e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaddec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffaddedb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffaddedc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffaddf7474
Build id: 3.10.0-1062.12.1.el7.x86_64
0xf4
8
? xfs_iunlock
xfs 0xffffffffc060e0c4
Build id: 3.10.0-1062.12.1.el7.x86_64
0xe4
9
? __vma_rb_erase
vmlinux 0xffffffffaddf7a31
Build id: 3.10.0-1062.12.1.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffffaddf9a75
Build id: 3.10.0-1062.12.1.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffffaddf9cb5
Build id: 3.10.0-1062.12.1.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffffaddfaf52
Build id: 3.10.0-1062.12.1.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffffae38dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffae38de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [icsfilesec:116627]
Modules linked in: tcp_lp fuse kbase(OE) xt_NFQUEUE nfnetlink_log nfnetlink nfring(OE) vmsecmod(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev joydev virtio_balloon sg i2c_piix4 pcspkr parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring serio_raw floppy virtio
drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmsec_nfq]
CPU: 8 PID: 116627 Comm: icsfilesec Kdump: loaded Tainted: GF          OEL ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9c1d36613150 ti: ffff9c2ce82c4000 task.ti: ffff9c2ce82c4000
RIP: 0010:[<ffffffffadd16a7a>]  [<ffffffffadd16a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c2ce82c7cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000adc7e2b0 RCX: ffff9c2cfce20518
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9c2ce82c7ce8 R08: ffff9c1e695dc800 R09: ffffffffadf81839
R10: ffff9c2cfd01f160 R11: fffff0548b9aec00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9c2ce82c7c60 R15: 00007ff6cee16000
FS:  00007ff69bfff700(0000) GS:ffff9c2cfd000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff6cee16000 CR3: 00000001db766000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffadc7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffadc7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaddec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaddedb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaddedc83>] tlb_finish_mmu+0x23/0x30
[<ffffffffaddf7474>] unmap_region+0xf4/0x140
[<ffffffffc060e0c4>] ? xfs_iunlock+0xe4/0x120 [xfs]
[<ffffffffaddf7a31>] ? __vma_rb_erase+0x121/0x220
[<ffffffffaddf9a75>] do_munmap+0x2a5/0x480
[<ffffffffaddf9cb5>] vm_munmap+0x65/0xb0
[<ffffffffaddfaf52>] SyS_munmap+0x22/0x30
[<ffffffffae38dede>] system_call_fastpath+0x25/0x2a
[<ffffffffae38de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 94 ae f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #332232
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff86516dda
Build id: 3.10.0-1160.21.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8647e868
Build id: 3.10.0-1160.21.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8647e8db
Build id: 3.10.0-1160.21.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff865ede26
Build id: 3.10.0-1160.21.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff865ef8ef
Build id: 3.10.0-1160.21.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff865efa13
Build id: 3.10.0-1160.21.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff865f9204
Build id: 3.10.0-1160.21.1.el7.x86_64
0xf4
8
? xfs_iunlock
xfs 0xffffffffc05813c4
Build id: 3.10.0-1160.21.1.el7.x86_64
0xe4
9
? __vma_rb_erase
vmlinux 0xffffffff865f97c1
Build id: 3.10.0-1160.21.1.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffff865fb7f5
Build id: 3.10.0-1160.21.1.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffff865fba35
Build id: 3.10.0-1160.21.1.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffff865fcd32
Build id: 3.10.0-1160.21.1.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffff86b95f92
Build id: 3.10.0-1160.21.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffff86b95ed5
Build id: 3.10.0-1160.21.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [pcsd:1213]
Modules linked in: ext4 mbcache jbd2 drbd(OE) squashfs loop iosf_mbi kvm_intel kvm snd_hda_codec_generic irqbypass snd_hda_intel crc32_pclmul snd_hda_codec ghash_clmulni_intel aesni_intel snd_hda_core snd_hwdep snd_seq lrw gf128mul sg joydev pcspkr glue_helper snd_seq_device snd_pcm ablk_helper snd_timer snd soundcore cryptd i2c_piix4 virtio_balloon nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_console virtio_blk qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci 8139too e1000 ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw virtio_pci virtio_ring virtio 8139cp drm_panel_orientation_quirks mii floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
CPU: 1 PID: 1213 Comm: pcsd Kdump: loaded Tainted: G           OE  ------------   3.10.0-1160.21.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ArchLinux 1.15.0-1 04/01/2014
task: ffff9c18a9dbb180 ti: ffff9c180328c000 task.ti: ffff9c180328c000
RIP: 0010:[<ffffffff86516dda>]  [<ffffffff86516dda>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c180328fcb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008647e6a0 RCX: ffff9c18bfc20400
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9c180328fce8 R08: ffff9c18b9f55800 R09: ffffffff86786419
R10: ffff9c18bfc9f160 R11: fffff975024f4900 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9c180328fc60 R15: 00007fe30b8bc000
FS:  00007fe2edc0d700(0000) GS:ffff9c18bfc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe30b8bc000 CR3: 000000013196c000 CR4: 0000000000060ee0
Call Trace:
[<ffffffff8647e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8647e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff865ede26>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff865ef8ef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff865efa13>] tlb_finish_mmu+0x23/0x30
[<ffffffff865f9204>] unmap_region+0xf4/0x140
[<ffffffffc05813c4>] ? xfs_iunlock+0xe4/0x120 [xfs]
[<ffffffff865f97c1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff865fb7f5>] do_munmap+0x2a5/0x480
[<ffffffff865fba35>] vm_munmap+0x65/0xb0
[<ffffffff865fcd32>] SyS_munmap+0x22/0x30
[<ffffffff86b95f92>] system_call_fastpath+0x25/0x2a
[<ffffffff86b95ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 0e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 15 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 dd 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #325695
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff84316c0e
Build id: 3.10.0-1160.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffff8427e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8427e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff843edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff843ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff843ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff843f90e4
Build id: 3.10.0-1160.el7.x86_64
0xf4
8
? xfs_iunlock
xfs 0xffffffffc06bd3c4
Build id: 3.10.0-1160.el7.x86_64
0xe4
9
? __vma_rb_erase
vmlinux 0xffffffff843f96a1
Build id: 3.10.0-1160.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffff843fb6d5
Build id: 3.10.0-1160.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffff843fb915
Build id: 3.10.0-1160.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffff843fcbb2
Build id: 3.10.0-1160.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffff84993f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 28s! [pool:128888]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 sb_edac snd_rawmidi snd_ac97_codec coretemp ac97_bus iosf_mbi crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device aesni_intel lrw snd_pcm gf128mul glue_helper snd_timer ablk_helper cryptd snd
ppdev vmw_balloon soundcore pcspkr joydev sg i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi libata mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 128888 Comm: pool Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff88fa919fa100 ti: ffff88fa0cfe4000 task.ti: ffff88fa0cfe4000
RIP: 0010:[<ffffffff84316c0e>]  [<ffffffff84316c0e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff88fa0cfe7cb0  EFLAGS: 00000202
RAX: 000000000000000a RBX: 000800008427e6a0 RCX: ffff8905bd8a0650
RDX: 000000000000000a RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88fa0cfe7ce8 R08: ffff88f724344800 R09: ffffffff84585f99
R10: ffff8905bd6df160 R11: ffffe08a53836a00 R12: 0000000000000282
R13: 0000000000000018 R14: ffff88fa0cfe7c20 R15: 0000000000000282
FS:  00007fe24a776700(0000) GS:ffff8905bd6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe25fad6000 CR3: 0000000110ba4000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8427e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8427e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff843edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff843ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff843ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff843f90e4>] unmap_region+0xf4/0x140
[<ffffffffc06bd3c4>] ? xfs_iunlock+0xe4/0x120 [xfs]
[<ffffffff843f96a1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff843fb6d5>] do_munmap+0x2a5/0x480
[<ffffffff843fb915>] vm_munmap+0x65/0xb0
[<ffffffff843fcbb2>] SyS_munmap+0x22/0x30
[<ffffffff84993f92>] system_call_fastpath+0x25/0x2a
Code: 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 f5 84 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #280635
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb4f170da
Build id: 3.10.0-1160.45.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb4e7e258
Build id: 3.10.0-1160.45.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb4e7e2cb
Build id: 3.10.0-1160.45.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb4fee676
Build id: 3.10.0-1160.45.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb4ff013f
Build id: 3.10.0-1160.45.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb4ff0263
Build id: 3.10.0-1160.45.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffb4ff9a54
Build id: 3.10.0-1160.45.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffb4ffa011
Build id: 3.10.0-1160.45.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffb4ffc045
Build id: 3.10.0-1160.45.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffb4ffc285
Build id: 3.10.0-1160.45.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffb4ffd582
Build id: 3.10.0-1160.45.1.el7.x86_64
0x22
12
tracesys
vmlinux 0xffffffffb5596226
Build id: 3.10.0-1160.45.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [Ruby-0-Thread-1:6355]
Modules linked in: vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) sb_edac sg i2c_piix4 iosf_mbi crc32_pclmul ghash_clmulni_intel virtio_balloon pcspkr aesni_intel lrw joydev gf128mul glue_helper ablk_helper cryptd ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom virtio_net net_failover virtio_scsi failover ata_generic pata_acpi bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix ttm drm libata crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci virtio_ring serio_raw virtio drm_panel_orientation_quirks floppy dm_mirror
dm_region_hash dm_log dm_mod
CPU: 1 PID: 6355 Comm: Ruby-0-Thread-1 Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
task: ffff8e5073f05280 ti: ffff8e4f40264000 task.ti: ffff8e4f40264000
RIP: 0010:[<ffffffffb4f170da>]  [<ffffffffb4f170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8e4f40267cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000c0000b4e7ddd0 RCX: ffff8e507fc20600
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8e4f40267ce8 R08: ffff8e4fbaee9000 R09: ffffffffb5186bf9
R10: ffff8e507fc5f160 R11: ffffefc7077d9600 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8e4f40267c60 R15: 00007f63d7501000
FS:  00007f629631d700(0000) GS:ffff8e507fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f63d7501000 CR3: 0000000234c0a000 CR4: 0000000000160fe0
Call Trace:
[<ffffffffb4e7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb4e7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb4fee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb4ff013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb4ff0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffb4ff9a54>] unmap_region+0xf4/0x140
[<ffffffffb4ffa011>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb4ffc045>] do_munmap+0x2a5/0x480
[<ffffffffb4ffc285>] vm_munmap+0x65/0xb0
[<ffffffffb4ffd582>] SyS_munmap+0x22/0x30
[<ffffffffb5596226>] tracesys+0xa6/0xcc
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 b5 b5 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #268919
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa0b16e4a
Build id: 3.10.0-1160.15.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa0a7e868
Build id: 3.10.0-1160.15.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa0a7e8db
Build id: 3.10.0-1160.15.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa0bede86
Build id: 3.10.0-1160.15.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa0bef94f
Build id: 3.10.0-1160.15.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa0befa73
Build id: 3.10.0-1160.15.2.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffa0bf9264
Build id: 3.10.0-1160.15.2.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffa0bf9821
Build id: 3.10.0-1160.15.2.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffa0bfb855
Build id: 3.10.0-1160.15.2.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffa0bfba95
Build id: 3.10.0-1160.15.2.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffa0bfcd92
Build id: 3.10.0-1160.15.2.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffa1194f92
Build id: 3.10.0-1160.15.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [UndertowEventHa:5658]
Modules linked in: gsch(OE) redirfs(OE) bmhook(OE) tmhook(OE) dsa_filter(POE) dsa_filter_hook(OE) tcp_lp rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 ppdev vmw_balloon sg joydev parport_pc parport sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul i2c_piix4
vmw_vmci glue_helper ablk_helper pcspkr cryptd ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi crct10dif_pclmul crct10dif_common mptscsih libata crc32c_intel serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: tmhook]
CPU: 6 PID: 5658 Comm: UndertowEventHa Kdump: loaded Tainted: P           OEL ------------   3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8ce6fc4bb180 ti: ffff8ce56ce8c000 task.ti: ffff8ce56ce8c000
RIP: 0010:[<ffffffffa0b16e4a>]  [<ffffffffa0b16e4a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8ce56ce8fcb0  EFLAGS: 00000202
RAX: 000000000000000b RBX: 00080000a0a7e6a0 RCX: ffff8ce70d8e06c8
RDX: 000000000000000b RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8ce56ce8fce8 R08: ffff8cdc21bac400 R09: ffffffffa0d86479
R10: ffff8ce70d79f160 R11: ffffd5b9c4ccfe00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8ce56ce8fc60 R15: 00007f58749a1000
FS:  00007f5869e28700(0000) GS:ffff8ce70d780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f58749a1000 CR3: 0000000a8aad2000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa0a7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa0a7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa0bede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa0bef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa0befa73>] tlb_finish_mmu+0x23/0x30
[<ffffffffa0bf9264>] unmap_region+0xf4/0x140
[<ffffffffa0bf9821>] ? __vma_rb_erase+0x121/0x220
[<ffffffffa0bfb855>] do_munmap+0x2a5/0x480
[<ffffffffa0bfba95>] vm_munmap+0x65/0xb0
[<ffffffffa0bfcd92>] SyS_munmap+0x22/0x30
[<ffffffffa1194f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 de 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 75 a1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #245149
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff94516dda
Build id: 3.10.0-1160.21.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9447e868
Build id: 3.10.0-1160.21.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9447e8db
Build id: 3.10.0-1160.21.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff945ede26
Build id: 3.10.0-1160.21.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff945ef8ef
Build id: 3.10.0-1160.21.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff945efa13
Build id: 3.10.0-1160.21.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff945f9204
Build id: 3.10.0-1160.21.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff945f97c1
Build id: 3.10.0-1160.21.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff945fb7f5
Build id: 3.10.0-1160.21.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff945fba35
Build id: 3.10.0-1160.21.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff945fcd32
Build id: 3.10.0-1160.21.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff94b95f92
Build id: 3.10.0-1160.21.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff94b95ed5
Build id: 3.10.0-1160.21.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [python3:23282]
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [mongod:23099]
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev pcspkr sg vmw_balloon vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt crc_t10dif fb_sys_fops crct10dif_generic ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw nfit vmxnet3 libnvdimm vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 23099 Comm: mongod Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffffa0af5b486300 ti: ffffa0b0e60f4000 task.ti: ffffa0b0e60f4000
RIP: 0010:[<ffffffff94516dda>]  [<ffffffff94516dda>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa0b0e60f7cb0  EFLAGS: 00000202
RAX: 0000000000000009 RBX: ffffffff9447e6a0 RCX: ffffa0b0ffce0728
RDX: 0000000000000009 RSI: 000000000000000c RDI: 0000000000000000
RBP: ffffa0b0e60f7ce8 R08: ffffa0afffc17c00 R09: ffffffff94786419
R10: ffffa0b0ffc1f160 R11: fffff5be889ad400 R12: 0000000000000286
R13: ffffa0b0e60f7c60 R14: 00007fe9f1036000 R15: 00007fe9f1035000
FS:  00007fe998421700(0000) GS:ffffa0b0ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe9f1035000 CR3: 000000014117e000 CR4: 00000000000407e0
Call Trace:
[<ffffffff9447e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9447e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff945ede26>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff945ef8ef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff945efa13>] tlb_finish_mmu+0x23/0x30
[<ffffffff945f9204>] unmap_region+0xf4/0x140
[<ffffffff945f97c1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff945fb7f5>] do_munmap+0x2a5/0x480
[<ffffffff945fba35>] vm_munmap+0x65/0xb0
[<ffffffff945fcd32>] SyS_munmap+0x22/0x30
[<ffffffff94b95f92>] system_call_fastpath+0x25/0x2a
[<ffffffff94b95ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 0e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 15 95 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 dd 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #160732
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff88b16a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff88a7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff88a7e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff88bec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff88bedb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff88bedc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff88bf7474
Build id: 3.10.0-1062.12.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff88bf7a31
Build id: 3.10.0-1062.12.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff88bf9a75
Build id: 3.10.0-1062.12.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff88bf9cb5
Build id: 3.10.0-1062.12.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff88bfaf52
Build id: 3.10.0-1062.12.1.el7.x86_64
0x22
12
tracesys
vmlinux 0xffffffff8918e172
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [pool-2996-threa:2893]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache xt_set ip_set_hash_net vxlan ip6_udp_tunnel udp_tunnel xt_nat xt_mark xfrm6_mode_tunnel xfrm4_mode_tunnel veth esp4 drbg ansi_cprng ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock crc32_pclmul
ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon sg parport_pc parport joydev pcspkr i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci ata_piix crct10dif_pclmul crct10dif_common crc32c_intel nfit libata serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 2893 Comm: pool-2996-threa Kdump: loaded Tainted: G        W      ------------ T 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8d8579f18000 ti: ffff8d84d0d64000 task.ti: ffff8d84d0d64000
RIP: 0010:[<ffffffff88b16a7a>]  [<ffffffff88b16a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8d84d0d67cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000088a7e2b0 RCX: ffff8d8cffc206a0
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8d84d0d67ce8 R08: ffff8d8634520800 R09: ffffffff88d81839
R10: ffff8d8cffd9f160 R11: fffff7ee00258800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8d84d0d67c60 R15: 00007effa9669000
FS:  00007eff4a9b4700(0000) GS:ffff8d8cffd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007effa9669000 CR3: 000000080ab46000 CR4: 00000000000007e0
Call Trace:
[<ffffffff88a7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff88a7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff88bec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff88bedb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff88bedc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff88bf7474>] unmap_region+0xf4/0x140
[<ffffffff88bf7a31>] ? __vma_rb_erase+0x121/0x220
[<ffffffff88bf9a75>] do_munmap+0x2a5/0x480
[<ffffffff88bf9cb5>] vm_munmap+0x65/0xb0
[<ffffffff88bfaf52>] SyS_munmap+0x22/0x30
[<ffffffff8918e172>] tracesys+0xa6/0xcc
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 74 89 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #138395
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff93116eea
Build id: 3.10.0-1160.24.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9307e868
Build id: 3.10.0-1160.24.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9307e8db
Build id: 3.10.0-1160.24.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff931edf56
Build id: 3.10.0-1160.24.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff931efa1f
Build id: 3.10.0-1160.24.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff931efb43
Build id: 3.10.0-1160.24.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff931f9334
Build id: 3.10.0-1160.24.1.el7.x86_64
0xf4
8
? blk_finish_plug
vmlinux 0xffffffff933577a4
Build id: 3.10.0-1160.24.1.el7.x86_64
0x14
9
? __vma_rb_erase
vmlinux 0xffffffff931f98f1
Build id: 3.10.0-1160.24.1.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffff931fb925
Build id: 3.10.0-1160.24.1.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffff931fbb65
Build id: 3.10.0-1160.24.1.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffff931fce62
Build id: 3.10.0-1160.24.1.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffff93795f92
Build id: 3.10.0-1160.24.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [Plex Media Serv:30321]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink arc4 md4 nls_utf8 cifs ccm dns_resolver nf_nat_ftp nf_conntrack_ftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper
ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt fb_sys_fops ttm crc_t10dif crct10dif_generic ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 30321 Comm: Plex Media Serv Kdump: loaded Tainted: G             L ------------   3.10.0-1160.24.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff99aebbcd6300 ti: ffff99ac99748000 task.ti: ffff99ac99748000
RIP: 0010:[<ffffffff93116eea>]  [<ffffffff93116eea>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff99ac9974bcb0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800009307e6a0 RCX: ffff99af7fca0650
RDX: 0000000000000002 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff99ac9974bce8 R08: ffff99acb9f42000 R09: ffffffff93386549
R10: ffff99af7fcdf160 R11: fffffecb4b4d3400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff99ac9974bc60 R15: 00007f8caa7fc000
FS:  00007f8c3e7fc700(0000) GS:ffff99af7fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f32741c1000 CR3: 000000041c7f8000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9307e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9307e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff931edf56>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff931efa1f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff931efb43>] tlb_finish_mmu+0x23/0x30
[<ffffffff931f9334>] unmap_region+0xf4/0x140
[<ffffffff933577a4>] ? blk_finish_plug+0x14/0x40
[<ffffffff931f98f1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff931fb925>] do_munmap+0x2a5/0x480
[<ffffffff931fbb65>] vm_munmap+0x65/0xb0
[<ffffffff931fce62>] SyS_munmap+0x22/0x30
[<ffffffff93795f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 d5 93 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #76889
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9af16e4a
Build id: 3.10.0-1160.15.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9ae7e868
Build id: 3.10.0-1160.15.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9ae7e8db
Build id: 3.10.0-1160.15.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9afede86
Build id: 3.10.0-1160.15.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9afef94f
Build id: 3.10.0-1160.15.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9afefa73
Build id: 3.10.0-1160.15.2.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff9aff9264
Build id: 3.10.0-1160.15.2.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff9aff9821
Build id: 3.10.0-1160.15.2.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff9affb855
Build id: 3.10.0-1160.15.2.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff9affba95
Build id: 3.10.0-1160.15.2.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff9affcd92
Build id: 3.10.0-1160.15.2.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff9b594f92
Build id: 3.10.0-1160.15.2.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff9b594ed5
Build id: 3.10.0-1160.15.2.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [sisidsdaemon:2946]
Modules linked in: binfmt_misc falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_10912(E) iptable_filter xt_DSCP iptable_mangle vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev vmw_balloon pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi ata_piix scsi_transport_spi nfit mptscsih libata serio_raw mptbase libnvdimm vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 2946 Comm: sisidsdaemon Kdump: loaded Tainted: P            EL ------------   3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff8be6133cb180 ti: ffff8be535d70000 task.ti: ffff8be535d70000
RIP: 0010:[<ffffffff9af16e4a>]  [<ffffffff9af16e4a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8be535d73cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffff9ae7e6a0 RCX: ffff8bf4fee20628
RDX: 0000000000000000 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff8be535d73ce8 R08: ffff8be663b43c00 R09: ffffffff9b186479
R10: ffff8bf4fee9f160 R11: fffffdedc599e000 R12: 0000000000000286
R13: ffff8be535d73c60 R14: 00007f6724e96000 R15: 00007f6724e95000
FS:  00007f670f7fe700(0000) GS:ffff8bf4fee80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6724e95000 CR3: 000000013278a000 CR4: 00000000000407e0
Call Trace:
[<ffffffff9ae7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9ae7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9afede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9afef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9afefa73>] tlb_finish_mmu+0x23/0x30
[<ffffffff9aff9264>] unmap_region+0xf4/0x140
[<ffffffff9aff9821>] ? __vma_rb_erase+0x121/0x220
[<ffffffff9affb855>] do_munmap+0x2a5/0x480
[<ffffffff9affba95>] vm_munmap+0x65/0xb0
[<ffffffff9affcd92>] SyS_munmap+0x22/0x30
[<ffffffff9b594f92>] system_call_fastpath+0x25/0x2a
[<ffffffff9b594ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 de 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 b5 9b f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #36375
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8391761a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8387e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8387e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff839ede36
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff839ef8ff
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff839efa23
Build id: 3.10.0-1127.19.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff839f9234
Build id: 3.10.0-1127.19.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff839f97f1
Build id: 3.10.0-1127.19.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff839fb825
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff839fba65
Build id: 3.10.0-1127.19.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff839fcd02
Build id: 3.10.0-1127.19.1.el7.x86_64
0x22
12
tracesys
vmlinux 0xffffffff83f93166
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [elasticsearch[G:2338]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus ppdev snd_seq snd_seq_device snd_pcm vmw_balloon snd_timer coretemp snd sg joydev pcspkr soundcore i2c_piix4 vmw_vmci parport_pc parport sch_fq ip_tables xfs libcrc32c
sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_intel ttm ata_piix drm mptspi nfit libata scsi_transport_spi libnvdimm serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 12 PID: 2338 Comm: elasticsearch[G Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9d261fb48000 ti: ffff9d2840430000 task.ti: ffff9d2840430000
RIP: 0010:[<ffffffff8391761a>]  [<ffffffff8391761a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9d2840433cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008387e730 RCX: ffff9d295d620790
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9d2840433ce8 R08: ffff9d22b4556400 R09: ffffffff83b851b9
R10: ffff9d295d91f160 R11: ffffedd4a071ba00 R12: 0000000000000282
R13: 0000000000000018 R14: ffff9d2840433c20 R15: 0000000000000282
FS:  00007f4fb2131700(0000) GS:ffff9d295d900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5161372000 CR3: 00000007aa6a6000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8387e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8387e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff839ede36>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff839ef8ff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff839efa23>] tlb_finish_mmu+0x23/0x30
[<ffffffff839f9234>] unmap_region+0xf4/0x140
[<ffffffff839f97f1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff839fb825>] do_munmap+0x2a5/0x480
[<ffffffff839fba65>] vm_munmap+0x65/0xb0
[<ffffffff839fcd02>] SyS_munmap+0x22/0x30
[<ffffffff83f93166>] tracesys+0xa6/0xcc
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 55 84 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #22724
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb9b16a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb9a7e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb9a7e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb9bec026
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb9bedaef
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb9bedc13
Build id: 3.10.0-1062.9.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffb9bf7404
Build id: 3.10.0-1062.9.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffb9bf79c1
Build id: 3.10.0-1062.9.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffb9bf9a05
Build id: 3.10.0-1062.9.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffb9bf9c45
Build id: 3.10.0-1062.9.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffb9bfaee2
Build id: 3.10.0-1062.9.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffba18dede
Build id: 3.10.0-1062.9.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [flussonic:26908]
Modules linked in: vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper vmw_balloon ablk_helper e1000e cryptd pcspkr ptp sg joydev pps_core parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif ahci drm crct10dif_generic libahci ata_piix nfit libata libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 26908 Comm: flussonic Kdump: loaded Tainted: G        W    L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
task: ffff8cea83d4e2a0 ti: ffff8ceb919b0000 task.ti: ffff8ceb919b0000
RIP: 0010:[<ffffffffb9b16a6a>]  [<ffffffffb9b16a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8ceb919b3cb0  EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00080000b9a7e2b0 RCX: ffff8cec7fde0678
RDX: 0000000000000007 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8ceb919b3ce8 R08: ffff8cebfcc10c00 R09: ffffffffb9d817e9
R10: ffff8cec7fd1f160 R11: fffff19c00231000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8ceb919b3c60 R15: 00007f980a1fd000
FS:  00007f992013f700(0000) GS:ffff8cec7fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9812644898 CR3: 00000001eff6a000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb9a7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb9a7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb9bec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb9bedaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb9bedc13>] tlb_finish_mmu+0x23/0x30
[<ffffffffb9bf7404>] unmap_region+0xf4/0x140
[<ffffffffb9bf79c1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb9bf9a05>] do_munmap+0x2a5/0x480
[<ffffffffb9bf9c45>] vm_munmap+0x65/0xb0
[<ffffffffb9bfaee2>] SyS_munmap+0x22/0x30
[<ffffffffba18dede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 74 ba f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #20137
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9c91761e
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffff9c87e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9c87e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9c9ede36
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9c9ef8ff
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9c9efa23
Build id: 3.10.0-1127.19.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff9c9f9234
Build id: 3.10.0-1127.19.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff9c9f97f1
Build id: 3.10.0-1127.19.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff9c9fb825
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff9c9fba65
Build id: 3.10.0-1127.19.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff9c9fcd02
Build id: 3.10.0-1127.19.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff9cf92ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 33s! [reader#1:1187]
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Modules linked in:
vsock_diag unix_diag af_packet_diag netlink_diag udp_diag tcp_diag inet_diag binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper pcspkr cryptd joydev sg parport_pc parport i2c_piix4 vmw_vmci CyProtectDrv(OE) ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod
crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi nfit libata scsi_transport_spi libnvdimm crct10dif_pclmul mptscsih crct10dif_common crc32c_intel serio_raw mptbase drm_panel_orientation_quirks vmxnet3 dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 1187 Comm: reader#1 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8be418d662a0 ti: ffff8be408f4c000 task.ti: ffff8be408f4c000
RIP: 0010:[<ffffffff9c91761e>]  [<ffffffff9c91761e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8be408f4fcb0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800009c87e730 RCX: ffff8be43fc9fc10
RDX: 0000000000000002 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8be408f4fce8 R08: ffff8bdd7fd55000 R09: ffffffff9cb851b9
R10: ffff8be43fc1f160 R11: fffff878a07c4a00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8be408f4fc60 R15: 00007f3e7ecb6000
FS:  00007f3e743cc700(0000) GS:ffff8be43fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dee20 CR3: 0000000035db8000 CR4: 00000000001607f0
Call Trace:
[<ffffffff9c87e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9c87e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9c9ede36>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9c9ef8ff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9c9efa23>] tlb_finish_mmu+0x23/0x30
[<ffffffff9c9f9234>] unmap_region+0xf4/0x140
[<ffffffff9c9f97f1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff9c9fb825>] do_munmap+0x2a5/0x480
[<ffffffff9c9fba65>] vm_munmap+0x65/0xb0
[<ffffffff9c9fcd02>] SyS_munmap+0x22/0x30
[<ffffffff9cf92ed2>] system_call_fastpath+0x25/0x2a
Code: 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 55 9d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #17342
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb3316c5e
Build id: 3.10.0-1160.6.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffb327e868
Build id: 3.10.0-1160.6.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb327e8db
Build id: 3.10.0-1160.6.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb33edd46
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb33ef80f
Build id: 3.10.0-1160.6.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb33ef933
Build id: 3.10.0-1160.6.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffb33f9124
Build id: 3.10.0-1160.6.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffb33f96e1
Build id: 3.10.0-1160.6.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffb33fb715
Build id: 3.10.0-1160.6.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffb33fb955
Build id: 3.10.0-1160.6.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffb33fcbf2
Build id: 3.10.0-1160.6.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffb3994f92
Build id: 3.10.0-1160.6.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffb3994ed5
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [psvc:26836]
Modules linked in: vmfsrtx_export(POE) vmfsrtx_bridge(POE) vmfsrtx_bridge_helper(OE) vmfsrtx_dev(POE) vmfsrtx_dev_helper(POE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd joydev pcspkr vmw_balloon sg parport_pc vmw_vmci
parport i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm ata_piix mptspi scsi_transport_spi mptscsih libata mptbase serio_raw vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: vmfsrtx_bridge_helper]
CPU: 2 PID: 26836 Comm: psvc Kdump: loaded Tainted: P        W  OEL ------------   3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9e2fc7bc2100 ti: ffff9e2f60bdc000 task.ti: ffff9e2f60bdc000
RIP: 0010:[<ffffffffb3316c5e>]  [<ffffffffb3316c5e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff9e2f60bdfcb0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffffffffb327e6a0 RCX: ffff9e4d7ef20628
RDX: 0000000000000004 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff9e2f60bdfce8 R08: ffff9e2f05f53c00 R09: ffffffffb3585fd9
R10: ffff9e4d7ee9f160 R11: ffffd88246fef800 R12: 0000000000000286
R13: ffff9e2f60bdfc60 R14: 00007f649bd13000 R15: 00007f649bd12000
FS:  00007f6472ffd700(0000) GS:ffff9e4d7ee80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f649bd12000 CR3: 00000001d13b0000 CR4: 00000000000407e0
Call Trace:
[<ffffffffb327e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb327e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb33edd46>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb33ef80f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb33ef933>] tlb_finish_mmu+0x23/0x30
[<ffffffffb33f9124>] unmap_region+0xf4/0x140
[<ffffffffb33f96e1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb33fb715>] do_munmap+0x2a5/0x480
[<ffffffffb33fb955>] vm_munmap+0x65/0xb0
[<ffffffffb33fcbf2>] SyS_munmap+0x22/0x30
[<ffffffffb3994f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb3994ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 f5 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #15522
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9e9150ca
Build id: 3.10.0-1062.1.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9e87d398
Build id: 3.10.0-1062.1.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9e87d40b
Build id: 3.10.0-1062.1.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9e9ea166
Build id: 3.10.0-1062.1.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9e9ebc2f
Build id: 3.10.0-1062.1.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9e9ebd53
Build id: 3.10.0-1062.1.2.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff9e9f5544
Build id: 3.10.0-1062.1.2.el7.x86_64
0xf4
8
? blk_finish_plug
vmlinux 0xffffffff9eb50cc4
Build id: 3.10.0-1062.1.2.el7.x86_64
0x14
9
? __vma_rb_erase
vmlinux 0xffffffff9e9f5b01
Build id: 3.10.0-1062.1.2.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffff9e9f7b45
Build id: 3.10.0-1062.1.2.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffff9e9f7d85
Build id: 3.10.0-1062.1.2.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffff9e9f9022
Build id: 3.10.0-1062.1.2.el7.x86_64
0x22
13
tracesys
vmlinux 0xffffffff9ef8d172
Build id: 3.10.0-1062.1.2.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [s1-agent:121307]
Modules linked in: btrfs raid6_pq xor msdos fat ext4 mbcache jbd2 udp_diag tcp_diag inet_diag binfmt_misc tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun bridge stp llc devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg i2c_piix4 vmw_vmci parport_pc
parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm crct10dif_pclmul crct10dif_common mptspi crc32c_intel scsi_transport_spi libata mptscsih serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 10 PID: 121307 Comm: s1-agent Kdump: loaded Tainted: G        W    L ------------   3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff98750347e2a0 ti: ffff98750c054000 task.ti: ffff98750c054000
RIP: 0010:[<ffffffff9e9150ca>]  [<ffffffff9e9150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff98750c057cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009e87d1d0 RCX: ffff987d1d620718
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff98750c057ce8 R08: ffff98767451e000 R09: ffffffff9eb7f629
R10: ffff987d1d89f160 R11: fffff8e619f1a400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff98750c057c60 R15: 00007f02848f2000
FS:  00007f0232ffd700(0000) GS:ffff987d1d880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000016c5238 CR3: 0000000108cde000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9e87d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9e87d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9e9ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9e9ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9e9ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffff9e9f5544>] unmap_region+0xf4/0x140
[<ffffffff9eb50cc4>] ? blk_finish_plug+0x14/0x40
[<ffffffff9e9f5b01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff9e9f7b45>] do_munmap+0x2a5/0x480
[<ffffffff9e9f7d85>] vm_munmap+0x65/0xb0
[<ffffffff9e9f9022>] SyS_munmap+0x22/0x30
[<ffffffff9ef8d172>] tracesys+0xa6/0xcc
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 54 9f f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #3683
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa6d16a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa6c7e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa6c7e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa6dec026
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa6dedaef
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa6dedc13
Build id: 3.10.0-1062.9.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffa6df7404
Build id: 3.10.0-1062.9.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffa6df79c1
Build id: 3.10.0-1062.9.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffa6df9a05
Build id: 3.10.0-1062.9.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffa6df9c45
Build id: 3.10.0-1062.9.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffa6dfaee2
Build id: 3.10.0-1062.9.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffa738dede
Build id: 3.10.0-1062.9.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [setroubleshootd:86700]
Modules linked in: xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables veth tcp_lp fuse br_netfilter tun devlink overlay(T) bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vmw_vmci vsock sunrpc snd_seq_midi snd_seq_midi_event coretemp ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 vmw_balloon snd_rawmidi snd_ac97_codec ac97_bus aesni_intel snd_seq lrw gf128mul glue_helper snd_seq_device snd_pcm ablk_helper cryptd snd_timer pcspkr joydev snd sg soundcore parport_pc parport pcc_cpufreq i2c_piix4 xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm crct10dif_pclmul crct10dif_common mptspi crc32c_intel scsi_transport_spi serio_raw drm mptscsih ata_piix e1000 nfit libnvdimm libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 3 PID: 86700 Comm: setroubleshootd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff96b282d4b150 ti: ffff96b29cc30000 task.ti: ffff96b29cc30000
RIP: 0010:[<ffffffffa6d16a6a>]  [<ffffffffa6d16a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff96b29cc33cb0  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000a6c7e2b0 RCX: ffff96b33a6604d8
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff96b29cc33ce8 R08: ffff96b3065c4c00 R09: ffffffffa6f817e9
R10: ffff96b33a6df160 R11: ffffd2cc40fd4000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff96b29cc33c60 R15: 00007f003cc4d000
FS:  00007f003cc30740(0000) GS:ffff96b33a6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f003cc4d000 CR3: 000000001cdae000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa6c7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa6c7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa6dec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa6dedaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa6dedc13>] tlb_finish_mmu+0x23/0x30
[<ffffffffa6df7404>] unmap_region+0xf4/0x140
[<ffffffffa6df79c1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffa6df9a05>] do_munmap+0x2a5/0x480
[<ffffffffa6df9c45>] vm_munmap+0x65/0xb0
[<ffffffffa6dfaee2>] SyS_munmap+0x22/0x30
[<ffffffffa738dede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 94 a7 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1617
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8ff1761a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8fe7e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8fe7e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8ffede36
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8ffef8ff
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8ffefa23
Build id: 3.10.0-1127.19.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff8fff9234
Build id: 3.10.0-1127.19.1.el7.x86_64
0xf4
8
? xfs_iunlock
xfs 0xffffffffc0384344
Build id: 3.10.0-1127.19.1.el7.x86_64
0xe4
9
? __vma_rb_erase
vmlinux 0xffffffff8fff97f1
Build id: 3.10.0-1127.19.1.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffff8fffb825
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffff8fffba65
Build id: 3.10.0-1127.19.1.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffff8fffcd02
Build id: 3.10.0-1127.19.1.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffff90592ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffff90592e15
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ns-slapd:5566]
Modules linked in: kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ppdev parport_pc joydev parport i2c_piix4 virtio_balloon pcspkr sch_fq_codel auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi qxl drm_kms_helper virtio_net net_failover failover syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common virtio_pci crc32c_intel floppy serio_raw virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 5566 Comm: ns-slapd Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20200516_175120-felixonmars2 04/01/2014
task: ffffa11678a08000 ti: ffffa11672244000 task.ti: ffffa11672244000
RIP: 0010:[<ffffffff8ff1761a>]  [<ffffffff8ff1761a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa11672247cb0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 000800008fe7e730 RCX: ffffa11800120428
RDX: 0000000000000004 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffffa11672247ce8 R08: ffffa1173cf55400 R09: ffffffff901851b9
R10: ffffa1180009f160 R11: ffffebf988d6e500 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa11672247c60 R15: 00007f08376b9000
FS:  00007f080ef67700(0000) GS:ffffa11800080000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f08376b9000 CR3: 00000000b8a12000 CR4: 0000000000020ee0
Call Trace:
[<ffffffff8fe7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8fe7e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8ffede36>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8ffef8ff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8ffefa23>] tlb_finish_mmu+0x23/0x30
[<ffffffff8fff9234>] unmap_region+0xf4/0x140
[<ffffffffc0384344>] ? xfs_iunlock+0xe4/0x120 [xfs]
[<ffffffff8fff97f1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff8fffb825>] do_munmap+0x2a5/0x480
[<ffffffff8fffba65>] vm_munmap+0x65/0xb0
[<ffffffff8fffcd02>] SyS_munmap+0x22/0x30
[<ffffffff90592ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff90592e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 b5 90 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1419
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8cd1519a
Build id: 3.10.0-1062.4.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8cc7d478
Build id: 3.10.0-1062.4.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8cc7d4eb
Build id: 3.10.0-1062.4.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8cdea756
Build id: 3.10.0-1062.4.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8cdec21f
Build id: 3.10.0-1062.4.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8cdec343
Build id: 3.10.0-1062.4.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff8cdf5b34
Build id: 3.10.0-1062.4.1.el7.x86_64
0xf4
8
? xfs_iunlock
xfs 0xffffffffc04370c4
Build id: 3.10.0-1062.4.1.el7.x86_64
0xe4
9
? __vma_rb_erase
vmlinux 0xffffffff8cdf60f1
Build id: 3.10.0-1062.4.1.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffff8cdf8135
Build id: 3.10.0-1062.4.1.el7.x86_64
0x2a5
11
vm_munmap
vmlinux 0xffffffff8cdf8375
Build id: 3.10.0-1062.4.1.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffff8cdf9612
Build id: 3.10.0-1062.4.1.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffff8d38bede
Build id: 3.10.0-1062.4.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [java:18159]
Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm irqbypass snd_ens1371 snd_rawmidi snd_ac97_codec ppdev ac97_bus crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device aesni_intel vmw_balloon snd_pcm lrw gf128mul glue_helper snd_timer ablk_helper cryptd snd soundcore pcspkr joydev nfit i2c_piix4 sg vmw_vmci libnvdimm parport_pc
parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel ata_piix serio_raw libata mptspi scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 18159 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-1062.4.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff8b43187f20e0 ti: ffff8b42e53e0000 task.ti: ffff8b42e53e0000
RIP: 0010:[<ffffffff8cd1519a>]  [<ffffffff8cd1519a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b42e53e3cb0  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000800008cc7d2b0 RCX: ffff8b456d660678
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8b42e53e3ce8 R08: ffff8b42b9cad400 R09: ffffffff8cf7fe99
R10: ffff8b456d71f160 R11: ffffe76d46c3a000 R12: 0000000000000282
R13: 0000000000000018 R14: ffff8b42e53e3c20 R15: 0000000000000282
FS:  00007f9f84c95700(0000) GS:ffff8b456d700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9f9c793000 CR3: 0000000200b98000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8cc7d478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8cc7d4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8cdea756>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8cdec21f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8cdec343>] tlb_finish_mmu+0x23/0x30
[<ffffffff8cdf5b34>] unmap_region+0xf4/0x140
[<ffffffffc04370c4>] ? xfs_iunlock+0xe4/0x120 [xfs]
[<ffffffff8cdf60f1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff8cdf8135>] do_munmap+0x2a5/0x480
[<ffffffff8cdf8375>] vm_munmap+0x65/0xb0
[<ffffffff8cdf9612>] SyS_munmap+0x22/0x30
[<ffffffff8d38bede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 6e 66 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d6 94 8d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 3d 66 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #826
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff949175ce
Build id: 3.10.0-1127.10.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffff9487e8f8
Build id: 3.10.0-1127.10.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9487e96b
Build id: 3.10.0-1127.10.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff949ede16
Build id: 3.10.0-1127.10.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff949ef8df
Build id: 3.10.0-1127.10.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff949efa03
Build id: 3.10.0-1127.10.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff949f9214
Build id: 3.10.0-1127.10.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffff949f97d1
Build id: 3.10.0-1127.10.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff949fb805
Build id: 3.10.0-1127.10.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff949fba45
Build id: 3.10.0-1127.10.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff949fcce2
Build id: 3.10.0-1127.10.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff94f92ed2
Build id: 3.10.0-1127.10.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff94f92e15
Build id: 3.10.0-1127.10.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [Timer-11:14741]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay iptable_filter ppdev pcspkr joydev parport_pc i2c_piix4 parport virtio_balloon ip_tables xfs libcrc32c ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_blk e1000 virtio_pci virtio_ring serio_raw floppy drm_panel_orientation_quirks virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 14741 Comm: Timer-11 Kdump: loaded Not tainted 3.10.0-1127.10.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff9ea846bfb150 ti: ffff9ea8a0878000 task.ti: ffff9ea8a0878000
RIP: 0010:[<ffffffff949175ce>]  [<ffffffff949175ce>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff9ea8a087bcb0  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffff9487e730 RCX: ffff9ea91fca0428
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9ea8a087bce8 R08: ffff9ea8fd6bdc00 R09: ffffffff94b85179
R10: ffff9ea91fd1f160 R11: ffffe943c0d6af00 R12: ffff9ea8a087bc80
R13: ffffffff94b850cf R14: ffff9ea8ffc01500 R15: 00007fb825b76000
FS:  00007fb81836e700(0000) GS:ffff9ea91fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb825b76000 CR3: 0000000180318000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9487e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9487e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff949ede16>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff949ef8df>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff949efa03>] tlb_finish_mmu+0x23/0x30
[<ffffffff949f9214>] unmap_region+0xf4/0x140
[<ffffffff949f97d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff949fb805>] do_munmap+0x2a5/0x480
[<ffffffff949fba45>] vm_munmap+0x65/0xb0
[<ffffffff949fcce2>] SyS_munmap+0x22/0x30
[<ffffffff94f92ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff94f92e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 10 55 95 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 6d 7b c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #377
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa3f175ca
Build id: 3.10.0-1127.13.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa3e7e8f8
Build id: 3.10.0-1127.13.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa3e7e96b
Build id: 3.10.0-1127.13.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa3fede16
Build id: 3.10.0-1127.13.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa3fef8df
Build id: 3.10.0-1127.13.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa3fefa03
Build id: 3.10.0-1127.13.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffa3ff9214
Build id: 3.10.0-1127.13.1.el7.x86_64
0xf4
8
? __vma_rb_erase
vmlinux 0xffffffffa3ff97d1
Build id: 3.10.0-1127.13.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffa3ffb805
Build id: 3.10.0-1127.13.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffa3ffba45
Build id: 3.10.0-1127.13.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffa3ffcce2
Build id: 3.10.0-1127.13.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffa4592ed2
Build id: 3.10.0-1127.13.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffa4592e15
Build id: 3.10.0-1127.13.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [C2 CompilerThre:27836]
Modules linked in: nf_conntrack_netlink nfnetlink binfmt_misc vboxnetadp(OE) vboxnetflt(OE) xt_conntrack iptable_filter xt_REDIRECT nf_nat_redirect vboxdrv(OE) iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack sg ppdev joydev pcspkr parport_pc virtio_balloon parport i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi virtio_net virtio_console net_failover virtio_scsi failover bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 27836 Comm: C2 CompilerThre Kdump: loaded Tainted: G           OEL ------------   3.10.0-1127.13.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
task: ffff8f9c51f69070 ti: ffff8f9aca03c000 task.ti: ffff8f9aca03c000
RIP: 0010:[<ffffffffa3f175ca>]  [<ffffffffa3f175ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8f9aca03fcb0  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00080000a3e7e730 RCX: ffff8f9d3fd9fc10
RDX: 0000000000000003 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8f9aca03fce8 R08: ffff8f9bffd52000 R09: ffffffffa4185179
R10: ffff8f9d3fc1f160 R11: ffffcf5d0811c400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8f9aca03fc60 R15: 00007fef13d89000
FS:  00007feee521c700(0000) GS:ffff8f9d3fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fef13d89000 CR3: 000000022b986000 CR4: 00000000000006f0
Call Trace:
[<ffffffffa3e7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa3e7e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa3fede16>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa3fef8df>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa3fefa03>] tlb_finish_mmu+0x23/0x30
[<ffffffffa3ff9214>] unmap_region+0xf4/0x140
[<ffffffffa3ff97d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffa3ffb805>] do_munmap+0x2a5/0x480
[<ffffffffa3ffba45>] vm_munmap+0x65/0xb0
[<ffffffffa3ffcce2>] SyS_munmap+0x22/0x30
[<ffffffffa4592ed2>] system_call_fastpath+0x25/0x2a
[<ffffffffa4592e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 de 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 10 b5 a4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ad 7b c4 00 eb b7 0f b6 4d cc 4c 89