NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [10.3.10.9-manag:70177]
Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache cuse fuse rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr ipmi_ssif sg mei_me i2c_i801 lpc_ich mei ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter acpi_pad knem(OE) ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) ast crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea
sysfillrect sysimgblt fb_sys_fops mlx5_core(OE) ttm drm igb mlxfw(OE) psample auxiliary(OE) dca devlink i2c_algo_bit mlx_compat(OE) drm_panel_orientation_quirks ptp pps_core ahci libahci libata nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod xpmem(OE)
CPU: 1 PID: 70177 Comm: 10.3.10.9-manag Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.el7.x86_64 #1
Hardware name: THTF Chaoqiang R628/T1DM-E2, BIOS T1DM56.03 09/28/2019
task: ffff92d10a6ae180 ti: ffff92ca79bf0000 task.ti: ffff92ca79bf0000
RIP: 0010:[<ffffffff8d65e26e>]  [<ffffffff8d65e26e>] igrab+0x1e/0x60
RSP: 0018:ffff92ca79bf3db8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: 002aaf8c64e23d7c RCX: ffff92e0566d5800
RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff92c6284b3788
RBP: ffff92ca79bf3dc8 R08: ffff92e0566d308c R09: 0000000000000001
R10: ffff92c85e7e38d8 R11: 7fffffffffffffff R12: ffff92ca79bf3d60
R13: 0000000000000001 R14: 00299245d0b150e8 R15: ffff92c85f65ac00
FS:  0000000000000000(0000) GS:ffff92c85f640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b4f9445f654 CR3: 000000169ca10000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffc0a4f7ba>] nfs_delegation_grab_inode+0x2a/0x50 [nfsv4]
[<ffffffffc0a50d5e>] nfs_reap_expired_delegations+0xbe/0x230 [nfsv4]
[<ffffffffc0a4cfad>] nfs4_state_manager+0x5fd/0x8d0 [nfsv4]
[<ffffffffc0a4d280>] ? nfs4_state_manager+0x8d0/0x8d0 [nfsv4]
[<ffffffffc0a4d29f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff8d4c1c31>] kthread+0xd1/0xe0
[<ffffffff8d4c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff8db74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff8d4c1b60>] ? insert_kthread_work+0x40/0x40
Code: 5c 89 d0 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 4c 8d a7 88 00 00 00 53 48 89 fb 4c 89 e7 e8 72 c1 50 00 <f6> 83 a0 00 00 00 30 75 19 f0 ff 83 30 01 00 00 4c 89 e7 c6 07 

NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [10.3.10.9-manag:186719]
Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache cuse fuse rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr joydev sg mei_me i2c_i801 lpc_ich mei ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter acpi_pad knem(OE) ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) ast crct10dif_pclmul drm_kms_helper crct10dif_common crc32c_intel syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm mlx5_core(OE) igb mlxfw(OE) ahci psample auxiliary(OE) dca devlink i2c_algo_bit mlx_compat(OE) ptp libahci drm_panel_orientation_quirks pps_core libata nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod xpmem(OE)
CPU: 19 PID: 186719 Comm: 10.3.10.9-manag Kdump: loaded Tainted: G           OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: THTF Chaoqiang R628/T1DM-E2, BIOS T1DM56.03 09/28/2019
task: ffff8eb8ef0a9040 ti: ffff8ebe3ca94000 task.ti: ffff8ebe3ca94000
RIP: 0010:[<ffffffffb4d6a3f0>]  [<ffffffffb4d6a3f0>] _raw_spin_lock+0x10/0x30
RSP: 0000:ffff8ebe3ca97db0  EFLAGS: 00000246
RAX: 0000000000000000 RBX: 002af4c7352fb311 RCX: ffff8ed68bee9800
RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff8ecc0ad97d20
RBP: ffff8ebe3ca97dc8 R08: ffff8ed68bee888c R09: 0000000000000001
R10: 00000000000003e1 R11: 0000000000000000 R12: ffff8ebe3ca97d60
R13: 0000000000000001 R14: 00298d0fe193efa7 R15: ffff8ed69da5ac00
FS:  0000000000000000(0000) GS:ffff8ed69da40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000782590 CR3: 00000012b7410000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffb485e26e>] ? igrab+0x1e/0x60
[<ffffffffc0b9d7ba>] nfs_delegation_grab_inode+0x2a/0x50 [nfsv4]
[<ffffffffc0b9ed5e>] nfs_reap_expired_delegations+0xbe/0x230 [nfsv4]
[<ffffffffc0b9afad>] nfs4_state_manager+0x5fd/0x8d0 [nfsv4]
[<ffffffffc0b9b280>] ? nfs4_state_manager+0x8d0/0x8d0 [nfsv4]
[<ffffffffc0b9b29f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffffb46c1c31>] kthread+0xd1/0xe0
[<ffffffffb46c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb4d74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb46c1b60>] ? insert_kthread_work+0x40/0x40
Code: 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 40 1b ff ff 5d c3 0f 1f 40 

NMI watchdog: BUG: soft lockup - CPU#31 stuck for 22s! [10.2.255.239-ma:109536]
Modules linked in: sctp_diag sctp dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag fuse nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat tun ebtable_filter ebtables xt_multiport 8021q garp mrp bridge stp llc zfs(POE) zunicode(POE) zavl(POE) icp(POE) openvswitch nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack libcrc32c ip6table_filter ip6_tables zcommon(POE) znvpair(POE) spl(OE) ipt_REJECT nf_reject_ipv4 iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm
iw_cm i40iw ib_core iTCO_wdt skx_edac edac_core coretemp iTCO_vendor_support intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev sg mei_me pcspkr shpchp mei lpc_ich i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler nfit wmi libnvdimm acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod sr_mod cdrom crc_t10dif crct10dif_generic uas usb_storage ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe mpt3sas drm crct10dif_pclmul crct10dif_common i40e crc32c_intel ahci igb libahci libata raid_class mdio scsi_transport_sas dca i2c_algo_bit ptp i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 31 PID: 109536 Comm: 10.2.255.239-ma Tainted: P           OE  ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Sugon CX50-G35/60P16B-TC4600T, BIOS 0KLSN218 05/22/2019
task: ffff880104911fa0 ti: ffff880137b74000 task.ti: ffff880137b74000
RIP: 0010:[<ffffffff816abc00>]  [<ffffffff816abc00>] _raw_spin_lock+0x10/0x30
RSP: 0018:ffff880137b77dd0  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88189a6e388c RCX: ffff88189a6e3000
RDX: 0000000000000001 RSI: 000000000000009b RDI: ffff881844409250
RBP: ffff880137b77de8 R08: ffff88189a6e388c R09: ffffffff81185487
R10: ffff882fddcd9c40 R11: ffffea006cc6d700 R12: ffff882fddcd9c40
R13: ffffea006cc6d700 R14: 0000000000000000 R15: ffff880137b77dc8
FS:  0000000000000000(0000) GS:ffff882fddcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f911e2a2490 CR3: 00000017baedb000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffffc0e6e3fb ffff88189a6e3560 ffff8817daf430c8 ffff880137b77e58
ffffffffc0e6f91e ffffffffc0e88340 ffff882f823d8180 ffff881844409200
0100000000000000 701c09cf5e8aebf1 0000000427f66c61 00000000b45726b6
Call Trace:
[<ffffffffc0e6e3fb>] ? nfs_delegation_grab_inode+0x1b/0x50 [nfsv4]
[<ffffffffc0e6f91e>] nfs_reap_expired_delegations+0xbe/0x220 [nfsv4]
[<ffffffffc0e6bc32>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc0e6bf00>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc0e6bf1f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 a4 2a ff ff 5d c3 0f 1f 40 

NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [192.168.20.225-:6414]
Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype xt_conntrack iptable_filter nf_conntrack_netlink cfg80211 rfkill ip_vs_rr xt_ipvs ip_vs xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_mark br_netfilter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay(T) ip_set nfnetlink bridge iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle vmw_vsock_vmci_transport vsock ppdev vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci shpchp i2c_piix4 nfit libnvdimm nfsd auth_rpcgss nfs_acl lockd grace ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel drm_kms_helper scsi_transport_iscsi syscopyarea sysfillrect ghash_clmulni_intel
sysimgblt fb_sys_fops ttm 8021q drm garp stp ata_piix llc aesni_intel ahci mrp lrw libahci gf128mul glue_helper ablk_helper cryptd serio_raw libata vmxnet3 vmw_pvscsi i2c_core sunrpc dm_mirror dm_region_hash dm_log dm_mod [last unloaded: iptable_filter]
CPU: 5 PID: 6414 Comm: 192.168.20.225- Tainted: G             L ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff880017fb9fa0 ti: ffff8805bc400000 task.ti: ffff8805bc400000
RIP: 0010:[<ffffffff8121d06e>]  [<ffffffff8121d06e>] igrab+0x2e/0x60
RSP: 0018:ffff8805bc403db8  EFLAGS: 00000206
RAX: 0000000000000000 RBX: ffff880bfeb07540 RCX: ffff880bf9b13000
RDX: 0000000000000001 RSI: 000000000000008f RDI: ffff8807798f8668
RBP: ffff8805bc403dc8 R08: ffff880bf9b1688c R09: 0000000000000000
R10: ffff880c3fd59c40 R11: ffffea000a782000 R12: 000000000000000b
R13: ffff880bbbf62200 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880c3fd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f98bd68a050 CR3: 00000000019f2000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff8804b66482d0 ffff8804b6648280 ffff8805bc403de8 ffffffffc06b53ca
ffff880bf9b13560 ffff880c05bc24c8 ffff8805bc403e58 ffffffffc06b68be
ffffffffc06cf3c0 ffff880bd3f619c0 ffff8804b6648280 0000000000000000
Call Trace:
[<ffffffffc06b53ca>] nfs_delegation_grab_inode+0x2a/0x50 [nfsv4]
[<ffffffffc06b68be>] nfs_reap_expired_delegations+0xbe/0x220 [nfsv4]
[<ffffffffc06b2c22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc06b2ef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc06b2f0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 66 90 55 48 89 e5 41 54 4c 8d a7 88 00 00 00 53 48 89 fb 4c 89 e7 e8 d2 ea 48 00 f6 83 a0 00 00 00 30 75 19 f0 ff 83 30 01 00 00 <4c> 89 e7 c6 07 00 66 66 66 90 48 89 d8 5b 41 5c 5d c3 4c 89 e7