Info

Permalink
Function
smp_call_function_many
First occurrence
2020-10-07
Last occurrence
2020-10-07
State
NEW
Components
kernel
Quality
-16

Statistics

Operating system Count
CentOS 7 1,320
Architecture Count
x86_64 1,320
Related packages Count
kernel 1,320
 0:3.10.0-1160.81.1.el7 659
 0:3.10.0-1062.el7 156
 0:3.10.0-1160.118.1.el7 120
 0:3.10.0-1160.el7 48
 0:3.10.0-1160.76.1.el7 40
 0:3.10.0-1160.102.1.el7 39
 0:3.10.0-1160.45.1.el7 36
 0:3.10.0-1160.88.1.el7 35
 0:3.10.0-1160.83.1.el7 28
 0:3.10.0-1160.62.1.el7 27
 0:3.10.0-1160.6.1.el7 19
 0:3.10.0-1160.49.1.el7 18
 0:3.10.0-1160.119.1.el7 15
 0:3.10.0-1160.71.1.el7 14
 0:3.10.0-1160.24.1.el7 8
 0:3.10.0-1160.2.2.el7 7
 0:3.10.0-1160.59.1.el7 7
 0:3.10.0-1160.108.1.el7 7
 0:3.10.0-1062.12.1.el7 6
 0:3.10.0-1160.90.1.el7 5
 0:3.10.0-1160.15.2.el7 4
 0:3.10.0-1160.31.1.el7 4
 0:3.10.0-1160.114.2.el7 3
 0:3.10.0-1062.9.1.el7 2
 0:3.10.0-1160.25.1.el7 2
 0:3.10.0-1160.53.1.el7 2
 0:3.10.0-1160.92.1.el7 2
 0:3.10.0-1160.95.1.el7 2
 0:3.10.0-1160.11.1.el7 1
 0:3.10.0-1160.21.1.el7 1
 0:3.10.0-1160.36.2.el7 1
 0:3.10.0-1160.66.1.el7 1
 0:3.10.0-1160.80.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1189545
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9411e25a
Build id: 3.10.0-1160.114.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff94081fa8
Build id: 3.10.0-1160.114.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9408201b
Build id: 3.10.0-1160.114.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff941fa09e
Build id: 3.10.0-1160.114.2.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffff941fbc1f
Build id: 3.10.0-1160.114.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff941fbd53
Build id: 3.10.0-1160.114.2.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff941ff29a
Build id: 3.10.0-1160.114.2.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff941f968d
Build id: 3.10.0-1160.114.2.el7.x86_64
0x49d
9
? pick_next_entity
vmlinux 0xffffffff940e8699
Build id: 3.10.0-1160.114.2.el7.x86_64
0xa9
10
? set_next_entity
vmlinux 0xffffffff940e918c
Build id: 3.10.0-1160.114.2.el7.x86_64
0x3c
11
system_call_fastpath
vmlinux 0xffffffff947c539a
Build id: 3.10.0-1160.114.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [code:4555]
Modules linked in: tcp_lp xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 crc32_pclmul ghash_clmulni_intel aesni_intel lrw snd_ens1371 ppdev gf128mul snd_rawmidi vmw_balloon
glue_helper snd_ac97_codec ablk_helper ac97_bus cryptd snd_seq snd_seq_device joydev pcspkr snd_pcm snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common crc32c_intel ttm ata_piix mptspi drm nfit scsi_transport_spi serio_raw libata mptscsih libnvdimm e1000 drm_panel_orientation_quirks mptbase fuse
CPU: 6 PID: 4555 Comm: code Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.114.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9121e74f5280 ti: ffff9121f2c08000 task.ti: ffff9121f2c08000
RIP: 0010:[<ffffffff9411e25a>]  [<ffffffff9411e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9121f2c0bc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000094081b10 RCX: ffff91227d0206c8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9121f2c0bca8 R08: ffff911db005cc00 R09: ffffffff9439c7d9
R10: ffff91227d19f160 R11: fffff64cd2093800 R12: 0000000000000282
R13: 0000000000000018 R14: ffff9121f2c0bbe0 R15: 0000000000000282
FS:  00007f72ca5e0dc0(0000) GS:ffff91227d180000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002c1c0257f0c0 CR3: 000000054b764000 CR4: 0000000000360fe0
Call Trace:
[<ffffffff94081fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9408201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff941fa09e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff941fbc1f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff941fbd53>] tlb_finish_mmu+0x23/0x40
[<ffffffff941ff29a>] zap_page_range+0xea/0x140
[<ffffffff941f968d>] SyS_madvise+0x49d/0xac0
[<ffffffff940e8699>] ? pick_next_entity+0xa9/0x190
[<ffffffff940e918c>] ? set_next_entity+0x3c/0x100
[<ffffffff947c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 d5 94 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1144187
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa691e25a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa6881fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa688201b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa69fa0ce
Build id: 3.10.0-1160.119.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa69fbc4f
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa69fbd83
Build id: 3.10.0-1160.119.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa69ff2ca
Build id: 3.10.0-1160.119.1.el7.x86_64
0xea
8
? kmem_cache_free
vmlinux 0xffffffffa6a33b6a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x1fa
9
SyS_madvise
vmlinux 0xffffffffa69f96bd
Build id: 3.10.0-1160.119.1.el7.x86_64
0x49d
10
system_call_fastpath
vmlinux 0xffffffffa6fc539a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mysqld:17011]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac vmw_vmci coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper vmw_balloon cryptd sg joydev pcspkr parport_pc parport i2c_piix4 nfsd auth_rpcgss
nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi libata scsi_transport_spi crct10dif_pclmul crct10dif_common crc32c_intel mptscsih serio_raw mptbase pcnet32 mii drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 17011 Comm: mysqld Tainted: G        W    L ------------   3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8cc135cfa100 ti: ffff8cc3edc78000 task.ti: ffff8cc3edc78000
RIP: 0010:[<ffffffffa691e25a>]  [<ffffffffa691e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8cc3edc7bc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffa6881b10 RCX: ffff8cc3ffc20600
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8cc3edc7bca8 R08: ffff8cc1363c0800 R09: ffffffffa6b9c7f9
R10: ffff8cc3ffc5f160 R11: ffffe63ccb05b400 R12: ffff8cc3edc7bc40
R13: ffffffffa6b9c72f R14: 0000000000000282 R15: 0000000000000000
FS:  00007f4b0fdf9700(0000) GS:ffff8cc3ffc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f40395a9000 CR3: 00000000ba904000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa6881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa688201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa69fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa69fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa69fbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffa69ff2ca>] zap_page_range+0xea/0x140
[<ffffffffa6a33b6a>] ? kmem_cache_free+0x1fa/0x220
[<ffffffffa69f96bd>] SyS_madvise+0x49d/0xac0
[<ffffffffa6fc539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 55 a7 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #974446
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb5b1e1ca
Build id: 3.10.0-1160.95.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb5a81fa8
Build id: 3.10.0-1160.95.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb5a8201b
Build id: 3.10.0-1160.95.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb5bfa00e
Build id: 3.10.0-1160.95.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb5bfbb8f
Build id: 3.10.0-1160.95.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb5bfbcc3
Build id: 3.10.0-1160.95.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb5bff20a
Build id: 3.10.0-1160.95.1.el7.x86_64
0xea
8
? __slab_free
vmlinux 0xffffffffb5c3323c
Build id: 3.10.0-1160.95.1.el7.x86_64
0x9c
9
? kmem_cache_free
vmlinux 0xffffffffb5c33aaa
Build id: 3.10.0-1160.95.1.el7.x86_64
0x1fa
10
SyS_madvise
vmlinux 0xffffffffb5bf95fd
Build id: 3.10.0-1160.95.1.el7.x86_64
0x49d
11
system_call_fastpath
vmlinux 0xffffffffb61c539a
Build id: 3.10.0-1160.95.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [lms::workflow:::111568]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event btusb snd_ens1371 btrtl snd_rawmidi btbcm btintel crc32_pclmul snd_ac97_codec ghash_clmulni_intel ac97_bus bluetooth ppdev snd_seq snd_seq_device aesni_intel lrw snd_pcm vmw_balloon gf128mul glue_helper ablk_helper cryptd rfkill joydev sg snd_timer pcspkr snd i2c_piix4 soundcore vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic
pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix libnvdimm mptspi scsi_transport_spi e1000 crct10dif_pclmul crct10dif_common libata mptscsih mptbase crc32c_intel serio_raw drm_panel_orientation_quirks fuse
CPU: 3 PID: 111568 Comm: lms::workflow:: Kdump: loaded Tainted: G             L ------------   3.10.0-1160.95.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9af7245a3180 ti: ffff9af5968f4000 task.ti: ffff9af5968f4000
RIP: 0010:[<ffffffffb5b1e1ca>]  [<ffffffffb5b1e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9af5968f7c70  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00080000b5a81b10 RCX: ffff9af735720650
RDX: 0000000000000004 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9af5968f7ca8 R08: ffff9af67a88b800 R09: ffffffffb5d9c749
R10: ffff9af7356df160 R11: fffff9d0c881e400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9af5968f7c20 R15: 0000000000000001
FS:  00007fcfd028d700(0000) GS:ffff9af7356c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcfd0287fd8 CR3: 00000002212b2000 CR4: 0000000000360fe0
Call Trace:
[<ffffffffb5a81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb5a8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb5bfa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffb5bfbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb5bfbcc3>] tlb_finish_mmu+0x23/0x40
[<ffffffffb5bff20a>] zap_page_range+0xea/0x140
[<ffffffffb5c3323c>] ? __slab_free+0x9c/0x2b0
[<ffffffffb5c33aaa>] ? kmem_cache_free+0x1fa/0x220
[<ffffffffb5bf95fd>] SyS_madvise+0x49d/0xac0
[<ffffffffb61c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 75 b6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #917268
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff83d1e23a
Build id: 3.10.0-1160.102.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff83c81fa8
Build id: 3.10.0-1160.102.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff83c8201b
Build id: 3.10.0-1160.102.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff83dfa07e
Build id: 3.10.0-1160.102.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffff83dfbbff
Build id: 3.10.0-1160.102.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff83dfbd33
Build id: 3.10.0-1160.102.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff83dff27a
Build id: 3.10.0-1160.102.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff83df966d
Build id: 3.10.0-1160.102.1.el7.x86_64
0x49d
9
? pick_next_entity
vmlinux 0xffffffff83ce8699
Build id: 3.10.0-1160.102.1.el7.x86_64
0xa9
10
? set_next_entity
vmlinux 0xffffffff83ce918c
Build id: 3.10.0-1160.102.1.el7.x86_64
0x3c
11
system_call_fastpath
vmlinux 0xffffffff843c539a
Build id: 3.10.0-1160.102.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 21s! [synology-backup:1097]
Modules linked in: binfmt_misc ipt_rpfilter xt_multiport xt_set iptable_raw ip_set_hash_ip ip_set_hash_net ip_set vxlan ip6_udp_tunnel udp_tunnel ipt_REJECT nf_reject_ipv4 ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_mark xt_comment ip6table_filter ip6table_mangle iptable_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables nf_tables veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_addrtype iptable_filter br_netfilter bridge stp llc overlay(T) ip_vs nf_conntrack vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi sb_edac snd_ac97_codec ac97_bus btusb btrtl ppdev iosf_mbi btbcm snd_seq crc32_pclmul btintel ghash_clmulni_intel
vmw_balloon snd_seq_device bluetooth snd_pcm aesni_intel lrw gf128mul glue_helper snd_timer ablk_helper cryptd snd joydev rfkill pcspkr sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul ata_piix crct10dif_common drm nfit crc32c_intel nvme libata e1000 serio_raw libnvdimm nvme_core drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod synosnap(OE)
CPU: 8 PID: 1097 Comm: synology-backup Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1160.102.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff96f65cdb2100 ti: ffff96f65b3e0000 task.ti: ffff96f65b3e0000
RIP: 0010:[<ffffffff83d1e23a>]  [<ffffffff83d1e23a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff96f65b3e3c70  EFLAGS: 00000202
RAX: 000000000000000b RBX: 0008000083c81b10 RCX: ffff96f65d8e0718
RDX: 000000000000000b RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff96f65b3e3ca8 R08: ffff96efac57d400 R09: ffffffff83f9c7b9
R10: ffff96f65d81f160 R11: ffffd61089347400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff96f65b3e3c20 R15: 0000000000000001
FS:  00007f6f7656a700(0000) GS:ffff96f65d800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c017be5000 CR3: 000000081a75e000 CR4: 00000000003607e0
Call Trace:
[<ffffffff83c81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff83c8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff83dfa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff83dfbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff83dfbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffff83dff27a>] zap_page_range+0xea/0x140
[<ffffffff83df966d>] SyS_madvise+0x49d/0xac0
[<ffffffff83ce8699>] ? pick_next_entity+0xa9/0x190
[<ffffffff83ce918c>] ? set_next_entity+0x3c/0x100
[<ffffffff843c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 4e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 19 95 84 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #880607
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbc11e07a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbc081f48
Build id: 3.10.0-1160.81.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbc081fbb
Build id: 3.10.0-1160.81.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbc1f9f0e
Build id: 3.10.0-1160.81.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbc1fba4f
Build id: 3.10.0-1160.81.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbc1fbb83
Build id: 3.10.0-1160.81.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffbc1ff0ca
Build id: 3.10.0-1160.81.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffbc1f949d
Build id: 3.10.0-1160.81.1.el7.x86_64
0x49d
9
system_call_fastpath
vmlinux 0xffffffffbc7c539a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [qtp2011482127-2:25607]
Modules linked in: af_packet_diag netlink_diag vsock_diag tcp_diag udp_diag inet_diag unix_diag xt_comment xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci parport_pc parport i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi nfit mptscsih crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libnvdimm mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash
dm_log dm_mod fuse
CPU: 3 PID: 25607 Comm: qtp2011482127-2 Tainted: G        W  OEL ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9901faa70000 ti: ffff9901c8800000 task.ti: ffff9901c8800000
RIP: 0010:[<ffffffffbc11e07a>]  [<ffffffffbc11e07a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9901c8803c70  EFLAGS: 00000202
RAX: 0000000000000005 RBX: 00080000bc081ab0 RCX: ffff9909ffd60650
RDX: 0000000000000005 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff9901c8803ca8 R08: ffff99032c412800 R09: ffffffffbc39c139
R10: ffff9909ffcdf160 R11: fffff9b84020d800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9901c8803c20 R15: 0000000000000001
FS:  00007f85cc228700(0000) GS:ffff9909ffcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc6edf5e000 CR3: 00000003ef32e000 CR4: 00000000000607e0
Call Trace:
[<ffffffffbc081f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbc081fbb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbc1f9f0e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffbc1fba4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbc1fbb83>] tlb_finish_mmu+0x23/0x40
[<ffffffffbc1ff0ca>] zap_page_range+0xea/0x140
[<ffffffffbc1f949d>] SyS_madvise+0x49d/0xac0
[<ffffffffbc7c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 d5 bc f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 19 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #845423
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa8b150ce
Build id: 3.10.0-1062.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffa8a7d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa8a7d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa8bea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa8bebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa8bebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa8beea5a
Build id: 3.10.0-1062.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffa8be976d
Build id: 3.10.0-1062.el7.x86_64
0x49d
9
tracesys
vmlinux 0xffffffffa918c172
Build id: 3.10.0-1062.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [Metrics-HealthC:25616]
Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm coretemp ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc i2c_piix4 parport vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul
crct10dif_common drm crc32c_intel ata_piix mptspi serio_raw scsi_transport_spi mptscsih libata mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 25616 Comm: Metrics-HealthC Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff956574ec8000 ti: ffff95654a7f4000 task.ti: ffff95654a7f4000
RIP: 0010:[<ffffffffa8b150ce>]  [<ffffffffa8b150ce>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff95654a7f7c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a8a7d1d0 RCX: ffff9565bfc20490
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff95654a7f7ca8 R08: ffff9564fcf10c00 R09: ffffffffa8d7f629
R10: ffff9565bfc9f160 R11: ffffc7ffc1747000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff95654a7f7c20 R15: 0000000000000001
FS:  00007f2612ce9700(0000) GS:ffff9565bfc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f268faae530 CR3: 00000001ee7a0000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa8a7d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa8a7d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa8bea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa8bebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa8bebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffa8beea5a>] zap_page_range+0xea/0x130
[<ffffffffa8be976d>] SyS_madvise+0x49d/0xac0
[<ffffffffa918c172>] tracesys+0xa6/0xcc
Code: 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 74 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #770594
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8b9150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8b87d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8b87d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8b9ea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8b9ebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8b9ebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff8b9eea5a
Build id: 3.10.0-1062.el7.x86_64
0xea
8
? do_sync_write
vmlinux 0xffffffff8ba47ba3
Build id: 3.10.0-1062.el7.x86_64
0x93
9
SyS_madvise
vmlinux 0xffffffff8b9e976d
Build id: 3.10.0-1062.el7.x86_64
0x49d
10
system_call_fastpath
vmlinux 0xffffffff8bf8bede
Build id: 3.10.0-1062.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [HTTPHandler:7500]
Modules linked in: binfmt_misc nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device snd_pcm vmw_balloon snd_timer snd iosf_mbi crc32_pclmul ghash_clmulni_intel sg soundcore aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit libnvdimm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi e1000 mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 7500 Comm: HTTPHandler Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff93d836f362a0 ti: ffff93d482300000 task.ti: ffff93d482300000
RIP: 0010:[<ffffffff8b9150ca>]  [<ffffffff8b9150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff93d482303c70  EFLAGS: 00000202
RAX: 0000000000000009 RBX: 000800008b87d1d0 RCX: ffff93dc3d860490
RDX: 0000000000000009 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff93d482303ca8 R08: ffff93cda9663400 R09: ffffffff8bb7f629
R10: ffff93dc3d65f160 R11: fffffd105f91bc00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff93d482303c20 R15: 0000000000000001
FS:  00007f9f0e836700(0000) GS:ffff93dc3d640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f42b2912980 CR3: 0000000fd2a78000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff8b87d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8b87d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8b9ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8b9ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8b9ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffff8b9eea5a>] zap_page_range+0xea/0x130
[<ffffffff8ba47ba3>] ? do_sync_write+0x93/0xe0
[<ffffffff8b9e976d>] SyS_madvise+0x49d/0xac0
[<ffffffff8bf8bede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 54 8c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #733157
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff90f1e1ca
Build id: 3.10.0-1160.90.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff90e81fa8
Build id: 3.10.0-1160.90.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff90e8201b
Build id: 3.10.0-1160.90.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff90ffa00e
Build id: 3.10.0-1160.90.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffff90ffbb8f
Build id: 3.10.0-1160.90.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff90ffbcc3
Build id: 3.10.0-1160.90.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff90fff20a
Build id: 3.10.0-1160.90.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff90ff95fd
Build id: 3.10.0-1160.90.1.el7.x86_64
0x49d
9
system_call_fastpath
vmlinux 0xffffffff915c539a
Build id: 3.10.0-1160.90.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [sdatad:2370]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink iscsi_target_mod target_core_mod sfs(OE) nbd(E) iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd crc32_pclmul soundcore ghash_clmulni_intel
aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg vmw_balloon pcspkr joydev parport_pc parport vmw_vmci i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata nfit crct10dif_pclmul crct10dif_common libnvdimm crc32c_intel mptspi scsi_transport_spi mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 2370 Comm: sdatad Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.90.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9d6b9f9cc200 ti: ffff9d69b2670000 task.ti: ffff9d69b2670000
RIP: 0010:[<ffffffff90f1e1ca>]  [<ffffffff90f1e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9d69b2673c70  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 0008000090e81b10 RCX: ffff9d6bb13a0600
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9d69b2673ca8 R08: ffff9d6a78ccb000 R09: ffffffff9119c669
R10: ffff9d6bb125f160 R11: ffffee448a4ca000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9d69b2673c20 R15: 0000000000000001
FS:  00007f5ca27fc700(0000) GS:ffff9d6bb1240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1c32a6d000 CR3: 00000000bb4c6000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffff90e81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff90e8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff90ffa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff90ffbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff90ffbcc3>] tlb_finish_mmu+0x23/0x40
[<ffffffff90fff20a>] zap_page_range+0xea/0x140
[<ffffffff90ff95fd>] SyS_madvise+0x49d/0xac0
[<ffffffff915c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 b5 91 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #715078
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa191e1ce
Build id: 3.10.0-1160.88.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffa1881fa8
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa188201b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa19fa00e
Build id: 3.10.0-1160.88.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa19fbb8f
Build id: 3.10.0-1160.88.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa19fbcc3
Build id: 3.10.0-1160.88.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa19ff20a
Build id: 3.10.0-1160.88.1.el7.x86_64
0xea
8
? user_path_at_empty
vmlinux 0xffffffffa1a6f342
Build id: 3.10.0-1160.88.1.el7.x86_64
0x72
9
SyS_madvise
vmlinux 0xffffffffa19f95fd
Build id: 3.10.0-1160.88.1.el7.x86_64
0x49d
10
system_call_fastpath
vmlinux 0xffffffffa1fc539a
Build id: 3.10.0-1160.88.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [splunkd:9773]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_15003(E) falcon_lsm_pinned_14812(E) falcon_lsm_pinned_14713(E) binfmt_misc ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter xt_nat vmw_vsock_vmci_transport vsock iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc i2c_piix4 parport vmw_vmci auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm ahci ata_piix libahci drm libata nfit crct10dif_pclmul crct10dif_common crc32c_intel serio_raw
libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
CPU: 6 PID: 9773 Comm: splunkd Kdump: loaded Tainted: P            E  ------------   3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff97022b2ac200 ti: ffff97062d824000 task.ti: ffff97062d824000
RIP: 0010:[<ffffffffa191e1ce>]  [<ffffffffa191e1ce>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff97062d827c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a1881b10 RCX: ffff97022fa20728
RDX: 0000000000000000 RSI: 0000000000000018 RDI: 0000000000000000
RBP: ffff97062d827ca8 R08: ffff96ff763b9c00 R09: ffffffffa1b9c669
R10: ffff97022fb9f160 R11: fffffb80043c8a00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff97062d827c20 R15: 0000000000000001
FS:  00007f2ad03ff700(0000) GS:ffff97022fb80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2abfce6008 CR3: 000000081f13e000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffa1881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa188201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa19fa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa19fbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa19fbcc3>] tlb_finish_mmu+0x23/0x40
[<ffffffffa19ff20a>] zap_page_range+0xea/0x140
[<ffffffffa1a6f342>] ? user_path_at_empty+0x72/0xc0
[<ffffffffa19f95fd>] SyS_madvise+0x49d/0xac0
[<ffffffffa1fc539a>] system_call_fastpath+0x25/0x2a
Code: 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 55 a2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #680692
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb651e1ca
Build id: 3.10.0-1160.88.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb6481fa8
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb648201b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb65fa00e
Build id: 3.10.0-1160.88.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb65fbb8f
Build id: 3.10.0-1160.88.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb65fbcc3
Build id: 3.10.0-1160.88.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb65ff20a
Build id: 3.10.0-1160.88.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffb65f95fd
Build id: 3.10.0-1160.88.1.el7.x86_64
0x49d
9
? pick_next_entity
vmlinux 0xffffffffb64e8619
Build id: 3.10.0-1160.88.1.el7.x86_64
0xa9
10
? set_next_entity
vmlinux 0xffffffffb64e8a0c
Build id: 3.10.0-1160.88.1.el7.x86_64
0x3c
11
system_call_fastpath
vmlinux 0xffffffffb6bc539a
Build id: 3.10.0-1160.88.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [pbzip2:6291]
Modules linked in: file_protector(OE) snapapi26(OE) ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ses gf128mul enclosure scsi_transport_sas glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg i2c_piix4 parport_pc vmw_vmci parport ip_tables xfs libcrc32c sr_mod cdrom ahci vmwgfx libahci ata_generic pata_acpi uas drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix sd_mod crc_t10dif drm crct10dif_generic usb_storage libata crct10dif_pclmul nfit crct10dif_common crc32c_intel serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod tun fuse [last unloaded: edd]
CPU: 7 PID: 6291 Comm: pbzip2 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff90e5c90c2100 ti: ffff90e57dc4c000 task.ti: ffff90e57dc4c000
RIP: 0010:[<ffffffffb651e1ca>]  [<ffffffffb651e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff90e57dc4fc70  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00080000b6481b10 RCX: ffff90e67f2a06f0
RDX: 0000000000000002 RSI: 000000000000000c RDI: 0000000000000000
RBP: ffff90e57dc4fca8 R08: ffff90d7d9603000 R09: ffffffffb679c669
R10: ffff90e67f3df160 R11: ffffedd7bfeac200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff90e57dc4fc20 R15: 0000000000000001
FS:  00007f93ac97e700(0000) GS:ffff90e67f3c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f934893cffc CR3: 0000000efd812000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb6481fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb648201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb65fa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffb65fbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb65fbcc3>] tlb_finish_mmu+0x23/0x40
[<ffffffffb65ff20a>] zap_page_range+0xea/0x140
[<ffffffffb65f95fd>] SyS_madvise+0x49d/0xac0
[<ffffffffb64e8619>] ? pick_next_entity+0xa9/0x190
[<ffffffffb64e8a0c>] ? set_next_entity+0x3c/0xe0
[<ffffffffb6bc539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 15 b7 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #669157
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa23150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa227d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa227d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa23ea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa23ebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa23ebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa23eea5a
Build id: 3.10.0-1062.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffa23e976d
Build id: 3.10.0-1062.el7.x86_64
0x49d
9
? update_curr
vmlinux 0xffffffffa22e3696
Build id: 3.10.0-1062.el7.x86_64
0x86
10
system_call_fastpath
vmlinux 0xffffffffa298bede
Build id: 3.10.0-1062.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [containerd-shim:12802]
Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel xt_multiport xt_nat xt_comment xt_mark nf_conntrack_netlink xt_addrtype tcp_lp nls_utf8 isofs fuse ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) br_netfilter ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc iosf_mbi vmw_balloon ppdev crc32_pclmul ghash_clmulni_intel
aesni_intel pcspkr lrw gf128mul glue_helper ablk_helper joydev cryptd sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel nfit e1000 mptspi libnvdimm scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 12802 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff8ff688f8a0e0 ti: ffff8ff6b379c000 task.ti: ffff8ff6b379c000
RIP: 0010:[<ffffffffa23150ca>]  [<ffffffffa23150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8ff6b379fc70  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00080000a227d1d0 RCX: ffff8ff6ba720628
RDX: 0000000000000004 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8ff6b379fca8 R08: ffff8ff6865c4c00 R09: ffffffffa257f629
R10: ffff8ff6ba7df160 R11: ffffe9a54191ea00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8ff6b379fc20 R15: ffffffffa33d9540
FS:  000000c000112890(0000) GS:ffff8ff6ba7c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0000fb010 CR3: 000000005e00c000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa227d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa227d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa23ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa23ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa23ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffa23eea5a>] zap_page_range+0xea/0x130
[<ffffffffa23e976d>] SyS_madvise+0x49d/0xac0
[<ffffffffa22e3696>] ? update_curr+0x86/0x1e0
[<ffffffffa298bede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 f4 a2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #605379
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaf316eea
Build id: 3.10.0-1160.24.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaf27e868
Build id: 3.10.0-1160.24.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffaf27e8db
Build id: 3.10.0-1160.24.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaf3edf56
Build id: 3.10.0-1160.24.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffaf3efa1f
Build id: 3.10.0-1160.24.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffaf3efb43
Build id: 3.10.0-1160.24.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffaf3f300a
Build id: 3.10.0-1160.24.1.el7.x86_64
0xea
8
? do_sync_write
vmlinux 0xffffffffaf44d313
Build id: 3.10.0-1160.24.1.el7.x86_64
0x93
9
SyS_madvise
vmlinux 0xffffffffaf3ed4fd
Build id: 3.10.0-1160.24.1.el7.x86_64
0x49d
10
? vfs_write
vmlinux 0xffffffffaf44dea8
Build id: 3.10.0-1160.24.1.el7.x86_64
0x168
11
system_call_fastpath
vmlinux 0xffffffffaf995f92
Build id: 3.10.0-1160.24.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 21s! [util.rb:23:3268]
Modules linked in: tcp_diag inet_diag cfg80211 rfkill sunrpc nfit libnvdimm iosf_mbi sg crc32_pclmul ppdev ghash_clmulni_intel parport_pc aesni_intel parport i2c_piix4 virtio_balloon lrw gf128mul pcspkr glue_helper ablk_helper joydev cryptd ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi ata_piix libata e1000 serio_raw virtio_pci crct10dif_pclmul crct10dif_common crc32c_intel virtio_ring floppy virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 3268 Comm: util.rb:23 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.24.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffffa03fed7a0000 ti: ffffa03ee2058000 task.ti: ffffa03ee2058000
RIP: 0010:[<ffffffffaf316eea>]  [<ffffffffaf316eea>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa03ee205bc70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000af27e6a0 RCX: ffffa046bfc60528
RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffffa03ee205bca8 R08: ffffa03ff1b7a800 R09: ffffffffaf586549
R10: ffffa046bfddf160 R11: ffffe8b31e08da00 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa03ee205bc20 R15: 0000000000000001
FS:  00007f3af7447700(0000) GS:ffffa046bfdc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001503cc0 CR3: 00000001683f6000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaf27e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaf27e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaf3edf56>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaf3efa1f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaf3efb43>] tlb_finish_mmu+0x23/0x30
[<ffffffffaf3f300a>] zap_page_range+0xea/0x130
[<ffffffffaf44d313>] ? do_sync_write+0x93/0xe0
[<ffffffffaf3ed4fd>] SyS_madvise+0x49d/0xac0
[<ffffffffaf44dea8>] ? vfs_write+0x168/0x1f0
[<ffffffffaf995f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 f5 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #566307
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff95916e4a
Build id: 3.10.0-1160.15.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9587e868
Build id: 3.10.0-1160.15.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9587e8db
Build id: 3.10.0-1160.15.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff959ede86
Build id: 3.10.0-1160.15.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff959ef94f
Build id: 3.10.0-1160.15.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff959efa73
Build id: 3.10.0-1160.15.2.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff959f2f3a
Build id: 3.10.0-1160.15.2.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff959ed42d
Build id: 3.10.0-1160.15.2.el7.x86_64
0x49d
9
? put_prev_entity
vmlinux 0xffffffff958e615b
Build id: 3.10.0-1160.15.2.el7.x86_64
0x8b
10
system_call_fastpath
vmlinux 0xffffffff95f94f92
Build id: 3.10.0-1160.15.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#30 stuck for 23s! [bwa-mem2:139722]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache uinput devlink binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14306(E) snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core qxl snd_hwdep ttm drm_kms_helper snd_seq syscopyarea snd_seq_device sysfillrect snd_pcm sysimgblt fb_sys_fops snd_timer
ppdev iTCO_wdt iTCO_vendor_support crc32_pclmul ghash_clmulni_intel snd drm aesni_intel parport_pc lrw gf128mul glue_helper ablk_helper joydev pcspkr nfsd lpc_ich virtio_balloon soundcore drm_panel_orientation_quirks i2c_i801 virtio_rng cryptd parport nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c virtio_net net_failover virtio_console virtio_blk failover ahci libahci crct10dif_pclmul virtio_pci crct10dif_common virtio_ring crc32c_intel libata serio_raw virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 30 PID: 139722 Comm: bwa-mem2 Kdump: loaded Tainted: P            EL ------------   3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.13.0-2.module+el8.4.0+534+4680a14e 04/01/2014
task: ffff9226f7f6e300 ti: ffff922930bac000 task.ti: ffff922930bac000
RIP: 0010:[<ffffffff95916e4a>]  [<ffffffff95916e4a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff922930bafc70  EFLAGS: 00000202
RAX: 0000000000000096 RBX: 000800009587e6a0 RCX: ffff940fadba08c0
RDX: 0000000000000096 RSI: 00000000000000ff RDI: 0000000000000080
RBP: ffff922930bafca8 R08: ffff9224a1740008 R09: ffffffff95b86479
R10: ffff940fabd9f160 R11: fffff9040938de00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff922930bafc20 R15: 0000000000000000
FS:  00007f40faffd700(0000) GS:ffff940fabd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f40faffb490 CR3: 0000000b5c7e2000 CR4: 0000000000760ee0
PKRU: 55555554
Call Trace:
[<ffffffff9587e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9587e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff959ede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff959ef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff959efa73>] tlb_finish_mmu+0x23/0x30
[<ffffffff959f2f3a>] zap_page_range+0xea/0x130
[<ffffffff959ed42d>] SyS_madvise+0x49d/0xac0
[<ffffffff958e615b>] ? put_prev_entity+0x8b/0x400
[<ffffffff95f94f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 de 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 55 96 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #502238
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8db170da
Build id: 3.10.0-1160.49.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8da7e258
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8da7e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8dbee676
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8dbf013f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8dbf0263
Build id: 3.10.0-1160.49.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff8dbf372a
Build id: 3.10.0-1160.49.1.el7.x86_64
0xea
8
? kmem_cache_free
vmlinux 0xffffffff8dc27022
Build id: 3.10.0-1160.49.1.el7.x86_64
0x1e2
9
SyS_madvise
vmlinux 0xffffffff8dbedc1d
Build id: 3.10.0-1160.49.1.el7.x86_64
0x49d
10
system_call_fastpath
vmlinux 0xffffffff8e195f92
Build id: 3.10.0-1160.49.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mysqld:89828]
Modules linked in: ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_ipv4 nf_log_common xt_CHECKSUM tun bridge stp ipt_MASQUERADE llc nf_nat_masquerade_ipv4 nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport ebtable_filter ebtables devlink ip6table_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi sb_edac snd_ac97_codec ac97_bus snd_seq coretemp snd_seq_device snd_pcm snd_timer snd iosf_mbi soundcore crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul
glue_helper ablk_helper ppdev cryptd joydev pcspkr sg vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix nfit libata e1000 mptspi libnvdimm scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 89828 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9bc6702ba100 ti: ffff9bc61b68c000 task.ti: ffff9bc61b68c000
RIP: 0010:[<ffffffff8db170da>]  [<ffffffff8db170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9bc61b68fc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008da7ddd0 RCX: ffff9bc674620600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9bc61b68fca8 R08: ffff9bc63b88b800 R09: ffffffff8dd86bf9
R10: ffff9bc67465f160 R11: ffffdc6c045c2c00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9bc61b68fc20 R15: 0000000000000001
FS:  00007fb9b9b14700(0000) GS:ffff9bc674640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6dc0082b81 CR3: 00000000b633e000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8da7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8da7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8dbee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8dbf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8dbf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff8dbf372a>] zap_page_range+0xea/0x130
[<ffffffff8dc27022>] ? kmem_cache_free+0x1e2/0x200
[<ffffffff8dbedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff8e195f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 75 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:37:crtc-0] flip_done timed out
Complete report #386895
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff96d170da
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff96c7e258
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff96c7e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff96dee676
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff96df013f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff96df0263
Build id: 3.10.0-1160.59.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff96df372a
Build id: 3.10.0-1160.59.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff96dedc1d
Build id: 3.10.0-1160.59.1.el7.x86_64
0x49d
9
system_call_fastpath
vmlinux 0xffffffff97399f92
Build id: 3.10.0-1160.59.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 26s! [splunkd:6323]
sched: RT throttling activated
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13207(E) binfmt_misc falcon_lsm_pinned_13109(E) ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc vmw_vmci i2c_piix4 parport auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm ahci libahci ata_piix nfit libata crct10dif_pclmul crct10dif_common libnvdimm crc32c_intel serio_raw vmxnet3 vmw_pvscsi
drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
CPU: 6 PID: 6323 Comm: splunkd Kdump: loaded Tainted: P            E  ------------   3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff92f06e29c200 ti: ffff92f00e45c000 task.ti: ffff92f00e45c000
RIP: 0010:[<ffffffff96d170da>]  [<ffffffff96d170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92f00e45fc70  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 0008000096c7ddd0 RCX: ffff92ec6fd20728
RDX: 0000000000000004 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff92f00e45fca8 R08: ffff92e9b6379c00 R09: ffffffff96f86c09
R10: ffff92ec6fd9f160 R11: ffffeba520b92400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff92f00e45fc20 R15: 0000000000000001
FS:  00007f7226dff700(0000) GS:ffff92ec6fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f721d0d5020 CR3: 000000082acd2000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff96c7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff96c7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff96dee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff96df013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff96df0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff96df372a>] zap_page_range+0xea/0x130
[<ffffffff96dedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff97399f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 95 97 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #365290
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8d116c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8d07e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8d07e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8d1edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8d1ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8d1ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff8d1f2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
8
? update_curr
vmlinux 0xffffffff8d0e4306
Build id: 3.10.0-1160.el7.x86_64
0x86
9
SyS_madvise
vmlinux 0xffffffff8d1ed2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
10
system_call_fastpath
vmlinux 0xffffffff8d793f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [Keep-Alive-Time:68834]
Modules linked in: veth xt_physdev xt_NFLOG nfnetlink_log xt_limit vxlan xt_set ip6_udp_tunnel udp_tunnel ip_set_hash_ip xt_multiport xt_statistic ip6t_MASQUERADE nf_nat_masquerade_ipv6 nf_conntrack_netlink xt_nat xt_mark xt_addrtype xt_comment ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip_set nfnetlink ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) br_netfilter bridge stp llc vmw_vsock_vmci_transport vsock sunrpc coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw vmw_balloon gf128mul
glue_helper ablk_helper cryptd sg joydev pcspkr parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata nfit crct10dif_pclmul crct10dif_common libnvdimm mptspi crc32c_intel e1000 scsi_transport_spi mptscsih mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 68834 Comm: Keep-Alive-Time Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8cbbd4473180 ti: ffff8cbbe4f8c000 task.ti: ffff8cbbe4f8c000
RIP: 0010:[<ffffffff8d116c0a>]  [<ffffffff8d116c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8cbbe4f8fc70  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800008d07e6a0 RCX: ffff8cbfa8ce06c8
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8cbbe4f8fca8 R08: ffff8cb26ef44400 R09: ffffffff8d385f99
R10: ffff8cbfa8d9f160 R11: ffffede6acad3200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8cbbe4f8fc20 R15: 0000000000000001
FS:  00007f87c7cfc700(0000) GS:ffff8cbfa8d80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c420145010 CR3: 0000000dacb50000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8d07e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8d07e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8d1edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8d1ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8d1ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff8d1f2dba>] zap_page_range+0xea/0x130
[<ffffffff8d0e4306>] ? update_curr+0x86/0x1e0
[<ffffffff8d1ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff8d793f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 d5 8d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #313101
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff85f16bda
Build id: 3.10.0-1160.31.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff85e7e258
Build id: 3.10.0-1160.31.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff85e7e2cb
Build id: 3.10.0-1160.31.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff85fee176
Build id: 3.10.0-1160.31.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff85fefc3f
Build id: 3.10.0-1160.31.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff85fefd63
Build id: 3.10.0-1160.31.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff85ff322a
Build id: 3.10.0-1160.31.1.el7.x86_64
0xea
8
? do_sync_write
vmlinux 0xffffffff8604d593
Build id: 3.10.0-1160.31.1.el7.x86_64
0x93
9
SyS_madvise
vmlinux 0xffffffff85fed71d
Build id: 3.10.0-1160.31.1.el7.x86_64
0x49d
10
? vfs_write
vmlinux 0xffffffff8604e128
Build id: 3.10.0-1160.31.1.el7.x86_64
0x168
11
system_call_fastpath
vmlinux 0xffffffff86595f92
Build id: 3.10.0-1160.31.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [logind:30953]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev ppdev pcspkr vmw_balloon vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel
drm ata_piix nfit mptspi libata scsi_transport_spi serio_raw libnvdimm e1000 mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 10 PID: 30953 Comm: logind Kdump: loaded Tainted: G        W      ------------ T 3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff949268fed280 ti: ffff9492e145c000 task.ti: ffff9492e145c000
RIP: 0010:[<ffffffff85f16bda>]  [<ffffffff85f16bda>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9492e145fc70  EFLAGS: 00000202
RAX: 000000000000001b RBX: ffffffff85e7ddd0 RCX: ffff94955ece0768
RDX: 000000000000001b RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff9492e145fca8 R08: ffff948eac068400 R09: ffffffff861866a9
R10: ffff94955e89f160 R11: fffffc5960646600 R12: 0000000000000286
R13: ffff9492e145fc20 R14: ffff949559234800 R15: 0000000000000001
FS:  00007f5780535100(0000) GS:ffff94955e880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f57815c57f0 CR3: 00000005a1f54000 CR4: 00000000001607e0
Call Trace:
[<ffffffff85e7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff85e7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff85fee176>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff85fefc3f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff85fefd63>] tlb_finish_mmu+0x23/0x30
[<ffffffff85ff322a>] zap_page_range+0xea/0x130
[<ffffffff8604d593>] ? do_sync_write+0x93/0xe0
[<ffffffff85fed71d>] SyS_madvise+0x49d/0xac0
[<ffffffff8604e128>] ? vfs_write+0x168/0x1f0
[<ffffffff86595f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 8a c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 b5 86 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 8a c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #305299
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9fd16a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9fc7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9fc7e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9fdec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9fdedb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9fdedc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff9fdf098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff9fdeb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [mysqld:97318]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [mysqld:97318]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [mysqld:97318]
Modules linked in: tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev virtio_balloon pcspkr sg joydev parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw
virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 97318 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9dafb5dfa0e0 ti: ffff9db32f6f4000 task.ti: ffff9db32f6f4000
RIP: 0010:[<ffffffff9fd16a7a>]  [<ffffffff9fd16a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9db32f6f7c70  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800009fc7e2b0 RCX: ffff9db3acee0428
RDX: 0000000000000003 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9db32f6f7ca8 R08: ffff9db0f9c1b000 R09: ffffffff9ff81839
R10: ffff9db3ace9f160 R11: fffff69750728c00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9db32f6f7c20 R15: 0000000000000001
FS:  00002b258710b700(0000) GS:ffff9db3ace80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000040f010 CR3: 0000000395f96000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9fc7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9fc7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9fdec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9fdedb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9fdedc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff9fdf098a>] zap_page_range+0xea/0x130
[<ffffffff9fdeb69d>] SyS_madvise+0x49d/0xac0
Complete report #243655
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8b116eea
Build id: 3.10.0-1160.25.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8b07e868
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8b07e8db
Build id: 3.10.0-1160.25.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8b1edf56
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8b1efa1f
Build id: 3.10.0-1160.25.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8b1efb43
Build id: 3.10.0-1160.25.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff8b1f300a
Build id: 3.10.0-1160.25.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff8b1ed4fd
Build id: 3.10.0-1160.25.1.el7.x86_64
0x49d
9
system_call_fastpath
vmlinux 0xffffffff8b795f92
Build id: 3.10.0-1160.25.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [1_scheduler:500]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ppdev joydev pcspkr vmw_balloon i2c_piix4 parport_pc parport vmw_vmci binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common e1000e syscopyarea sysfillrect
crc32c_intel serio_raw sysimgblt ahci fb_sys_fops ttm libahci ptp pps_core drm nfit ata_piix vmw_pvscsi libnvdimm libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 500 Comm: 1_scheduler Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff94d3fe49d280 ti: ffff94d460060000 task.ti: ffff94d460060000
RIP: 0010:[<ffffffff8b116eea>]  [<ffffffff8b116eea>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff94d460063c70  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800008b07e6a0 RCX: ffff94d9ffc9fc10
RDX: 0000000000000002 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff94d460063ca8 R08: ffff94d33fd4f800 R09: ffffffff8b386549
R10: ffff94d9ffc1f160 R11: fffffc7f1aaa4400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff94d460063c20 R15: 0000000000000001
FS:  00007f64d0abf700(0000) GS:ffff94d9ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8225ca89e5 CR3: 000000055954a000 CR4: 00000000003607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8b07e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8b07e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8b1edf56>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8b1efa1f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8b1efb43>] tlb_finish_mmu+0x23/0x30
[<ffffffff8b1f300a>] zap_page_range+0xea/0x130
[<ffffffff8b1ed4fd>] SyS_madvise+0x49d/0xac0
[<ffffffff8b795f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 d5 8b f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #151928
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb5716e4a
Build id: 3.10.0-1160.15.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb567e868
Build id: 3.10.0-1160.15.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb567e8db
Build id: 3.10.0-1160.15.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb57ede86
Build id: 3.10.0-1160.15.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb57ef94f
Build id: 3.10.0-1160.15.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb57efa73
Build id: 3.10.0-1160.15.2.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb57f2f3a
Build id: 3.10.0-1160.15.2.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffb57ed42d
Build id: 3.10.0-1160.15.2.el7.x86_64
0x49d
9
? __check_object_size
vmlinux 0xffffffffb584a8da
Build id: 3.10.0-1160.15.2.el7.x86_64
0x1ca
10
system_call_fastpath
vmlinux 0xffffffffb5d94f92
Build id: 3.10.0-1160.15.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [util.rb:23:7096]
Modules linked in: nf_conntrack_netlink nfnetlink xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE br_netfilter nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device crc32_pclmul snd_pcm ghash_clmulni_intel snd_timer aesni_intel snd lrw gf128mul glue_helper ablk_helper cryptd soundcore ppdev pcspkr vmw_balloon sg joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common nfit crc32c_intel mptspi libnvdimm scsi_transport_spi ata_piix mptscsih libata serio_raw mptbase e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 7096 Comm: util.rb:23 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff888280601080 ti: ffff8882ba534000 task.ti: ffff8882ba534000
RIP: 0010:[<ffffffffb5716e4a>]  [<ffffffffb5716e4a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8882ba537c70  EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00080000b567e6a0 RCX: ffff8883b57e0600
RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8882ba537ca8 R08: ffff8882fe34b000 R09: ffffffffb5986479
R10: ffff8883b565f160 R11: ffffd0228097be00 R12: 0000000000000282
R13: 0000000000000018 R14: ffff8882ba537be0 R15: 0000000000000282
FS:  00007f3e061fe700(0000) GS:ffff8883b5640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3e0ee7a000 CR3: 00000001b0ba2000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb567e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb567e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb57ede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb57ef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb57efa73>] tlb_finish_mmu+0x23/0x30
[<ffffffffb57f2f3a>] zap_page_range+0xea/0x130
[<ffffffffb57ed42d>] SyS_madvise+0x49d/0xac0
[<ffffffffb584a8da>] ? __check_object_size+0x1ca/0x250
[<ffffffffb5d94f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 de 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 35 b6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #108919
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff96716a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9667e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9667e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff967ec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff967edb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff967edc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff967f098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff967eb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
9
system_call_fastpath
vmlinux 0xffffffff96d8dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [grafana-server:4972]
Modules linked in: binfmt_misc tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev pcspkr sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod
cdrom sd_mod crc_t10dif crct10dif_generic ata_generic vmwgfx drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi drm scsi_transport_spi ata_piix e1000 mptscsih crct10dif_pclmul crct10dif_common mptbase crc32c_intel libata serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 4972 Comm: grafana-server Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8a792eae41c0 ti: ffff8a79384f8000 task.ti: ffff8a79384f8000
RIP: 0010:[<ffffffff96716a7a>]  [<ffffffff96716a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a79384fbc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009667e2b0 RCX: ffff8a7971c204b0
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8a79384fbca8 R08: ffff8a796e95d000 R09: ffffffff96981839
R10: ffff8a7971d1f160 R11: ffffd855c01b5200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8a79384fbc20 R15: 0000000000000001
FS:  00007f46effff700(0000) GS:ffff8a7971d00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f46e009d868 CR3: 00000000384d0000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9667e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9667e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff967ec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff967edb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff967edc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff967f098a>] zap_page_range+0xea/0x130
[<ffffffff967eb69d>] SyS_madvise+0x49d/0xac0
[<ffffffff96d8dede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 34 97 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #67634
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8c516c5a
Build id: 3.10.0-1160.6.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8c47e868
Build id: 3.10.0-1160.6.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8c47e8db
Build id: 3.10.0-1160.6.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8c5edd46
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8c5ef80f
Build id: 3.10.0-1160.6.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8c5ef933
Build id: 3.10.0-1160.6.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff8c5f2dfa
Build id: 3.10.0-1160.6.1.el7.x86_64
0xea
8
? kmem_cache_free
vmlinux 0xffffffff8c626682
Build id: 3.10.0-1160.6.1.el7.x86_64
0x1e2
9
SyS_madvise
vmlinux 0xffffffff8c5ed2ed
Build id: 3.10.0-1160.6.1.el7.x86_64
0x49d
10
system_call_fastpath
vmlinux 0xffffffff8cb94f92
Build id: 3.10.0-1160.6.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mysqld:94002]
IPv4: martian source 10.1.134.255 from 10.1.134.58, on dev ens224
ll header: 00000000: ff ff ff ff ff ff 00 0c 29 f4 e8 4f 08 00        ........)..O..
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper ata_piix syscopyarea ahci sysfillrect sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common crc32c_intel drm libata nfit libnvdimm serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: CyProtectDrv]
CPU: 1 PID: 94002 Comm: mysqld Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8e7ce178d280 ti: ffff8e7c858e0000 task.ti: ffff8e7c858e0000
RIP: 0010:[<ffffffff8c516c5a>]  [<ffffffff8c516c5a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8e7c858e3c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008c47e6a0 RCX: ffff8e839d620600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8e7c858e3ca8 R08: ffff8e7cf2f5c000 R09: ffffffff8c785fd9
R10: ffff8e839d65f160 R11: fffff793119b0400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8e7c858e3c20 R15: 0000000000000000
FS:  00007f383755b700(0000) GS:ffff8e839d640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3850084c98 CR3: 00000003b2d70000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8c47e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8c47e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8c5edd46>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8c5ef80f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8c5ef933>] tlb_finish_mmu+0x23/0x30
[<ffffffff8c5f2dfa>] zap_page_range+0xea/0x130
[<ffffffff8c626682>] ? kmem_cache_free+0x1e2/0x200
[<ffffffff8c5ed2ed>] SyS_madvise+0x49d/0xac0
[<ffffffff8cb94f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 15 8d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #24727
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9b916c5a
Build id: 3.10.0-1160.6.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9b87e868
Build id: 3.10.0-1160.6.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9b87e8db
Build id: 3.10.0-1160.6.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9b9edd46
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9b9ef80f
Build id: 3.10.0-1160.6.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9b9ef933
Build id: 3.10.0-1160.6.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff9b9f2dfa
Build id: 3.10.0-1160.6.1.el7.x86_64
0xea
8
? handle_mm_fault
vmlinux 0xffffffff9b9f5e4d
Build id: 3.10.0-1160.6.1.el7.x86_64
0xbfd
9
SyS_madvise
vmlinux 0xffffffff9b9ed2ed
Build id: 3.10.0-1160.6.1.el7.x86_64
0x49d
10
system_call_fastpath
vmlinux 0xffffffff9bf94f92
Build id: 3.10.0-1160.6.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [mysqld:88644]
Modules linked in: nls_utf8 isofs bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 vmw_balloon aesni_intel snd_rawmidi snd_ac97_codec lrw gf128mul ac97_bus glue_helper ablk_helper cryptd snd_seq snd_seq_device snd_pcm pcspkr joydev snd_timer snd soundcore sg i2c_piix4 vmw_vmci parport_pc parport xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul ata_piix crct10dif_common drm crc32c_intel nfit mptspi libata scsi_transport_spi libnvdimm mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 11 PID: 88644 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff96cdbaf96300 ti: ffff96ca2e110000 task.ti: ffff96ca2e110000
RIP: 0010:[<ffffffff9b916c5a>]  [<ffffffff9b916c5a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff96ca2e113c70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000800009b87e6a0 RCX: ffffccd3fde40480
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff96ca2e113ca8 R08: ffff96c9e36c9000 R09: ffffffff9bb85fd9
R10: ffff96cded8df160 R11: fffff133cbaf0c00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff96ca2e113c20 R15: 0000000000000001
FS:  00007fc470360700(0000) GS:ffff96cded8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc470359fa0 CR3: 00000000b9dca000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9b87e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9b87e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9b9edd46>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9b9ef80f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9b9ef933>] tlb_finish_mmu+0x23/0x30
[<ffffffff9b9f2dfa>] zap_page_range+0xea/0x130
[<ffffffff9b9f5e4d>] ? handle_mm_fault+0xbfd/0xfb0
[<ffffffff9b9ed2ed>] SyS_madvise+0x49d/0xac0
[<ffffffff9bf94f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 55 9c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #24147
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbbf16c0a
Build id: 3.10.0-1160.2.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbbe7e868
Build id: 3.10.0-1160.2.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbbe7e8db
Build id: 3.10.0-1160.2.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbbfedd06
Build id: 3.10.0-1160.2.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbbfef7cf
Build id: 3.10.0-1160.2.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbbfef8f3
Build id: 3.10.0-1160.2.2.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffbbff2dba
Build id: 3.10.0-1160.2.2.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffbbfed2ad
Build id: 3.10.0-1160.2.2.el7.x86_64
0x49d
9
? schedule
vmlinux 0xffffffffbc5871e9
Build id: 3.10.0-1160.2.2.el7.x86_64
0x29
10
system_call_fastpath
vmlinux 0xffffffffbc593f92
Build id: 3.10.0-1160.2.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [sqlservr:75114]
Modules linked in: tcp_lp binfmt_misc xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec ppdev ac97_bus snd_seq aesni_intel snd_seq_device snd_pcm lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd pcspkr snd_timer joydev snd sg soundcore vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm libnvdimm ata_piix libata e1000 crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi mptscsih serio_raw drm_panel_orientation_quirks mptbase dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: cvblk]
CPU: 3 PID: 75114 Comm: sqlservr Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-1160.2.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff92070aae9080 ti: ffff9207c1114000 task.ti: ffff9207c1114000
RIP: 0010:[<ffffffffbbf16c0a>]  [<ffffffffbbf16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9207c1117c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000bbe7e6a0 RCX: ffff9207f9620650
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9207c1117ca8 R08: ffff9207f0fe3800 R09: ffffffffbc185f99
R10: ffff9207f96df160 R11: ffffcb8840787000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9207c1117c20 R15: 0000000000000001
FS:  00007f26c26bf700(0000) GS:ffff9207f96c0000(0000) knlGS:0000000121870000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000682696c08 CR3: 0000000078a92000 CR4: 00000000003607e0
Call Trace:
[<ffffffffbbe7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbbe7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbbfedd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbbfef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbbfef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffbbff2dba>] zap_page_range+0xea/0x130
[<ffffffffbbfed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffbc5871e9>] ? schedule+0x29/0x70
[<ffffffffbc593f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 b5 bc f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #6337
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff84d16a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff84c7e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff84c7e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff84dec026
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff84dedaef
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff84dedc13
Build id: 3.10.0-1062.9.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff84df091a
Build id: 3.10.0-1062.9.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff84deb62d
Build id: 3.10.0-1062.9.1.el7.x86_64
0x49d
9
tracesys
vmlinux 0xffffffff8538e172
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [C2 CompilerThre:7762]
Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel
lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nfit crct10dif_pclmul libata mptspi crct10dif_common crc32c_intel libnvdimm scsi_transport_spi serio_raw mptscsih mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 7762 Comm: C2 CompilerThre Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff97c6b9fc9070 ti: ffff97c2e7a50000 task.ti: ffff97c2e7a50000
RIP: 0010:[<ffffffff84d16a6a>]  [<ffffffff84d16a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff97c2e7a53c70  EFLAGS: 00000202
RAX: 0000000000000005 RBX: ffffffff84c7e2b0 RCX: ffff97c6ffd60678
RDX: 0000000000000005 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff97c2e7a53ca8 R08: ffff97c0344b1400 R09: ffffffff84f817e9
R10: ffff97c6ffd1f160 R11: ffffcd8589688000 R12: ffff97c2e7a53c40
R13: ffffffff84f8173f R14: ffff97c15ab20000 R15: 0000000000000001
FS:  00007fced84f4700(0000) GS:ffff97c6ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fce6d134fd0 CR3: 00000000bb81c000 CR4: 00000000001607e0
Call Trace:
[<ffffffff84c7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff84c7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff84dec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff84dedaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff84dedc13>] tlb_finish_mmu+0x23/0x30
[<ffffffff84df091a>] zap_page_range+0xea/0x130
[<ffffffff84deb62d>] SyS_madvise+0x49d/0xac0
[<ffffffff8538e172>] tracesys+0xa6/0xcc
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 94 85 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89