BUG: soft lockup in smp_call

BUG: soft lockup in smp_call_function
Status: upstream: reported C repro on 2020/07/12 23:02
Reported-by: syzbot+cce3691658bef1b12ac9@syzkaller.appspotmail.com
First crash: 465d, last: 3h52m

Cause bisection: introduced by (bisect log) :
commit 5a781ccbd19e4664babcbe4b4ead7aa2b9283d22
Author: Vinicius Costa Gomes <vinicius.gomes@intel.com>
Date: Sat Sep 29 00:59:43 2018 +0000

tc: Add support for configuring the taprio scheduler

Crash: BUG: soft lockup in smp_call_function (log)
Repro: C syz .config

similar bugs (1):
Kernel	Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
upstream	INFO: rcu detected stall in smp_call_function	C			107	190d	445d	22/22	fixed on 2021/04/13 06:26

Sample crash report:

watchdog: BUG: soft lockup - CPU#0 stuck for 118s! [syz-executor433:8673]
Modules linked in:
irq event stamp: 28124
hardirqs last  enabled at (28123): [<ffffffff89400c02>] asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
hardirqs last disabled at (28124): [<ffffffff892a1efb>] sysvec_apic_timer_interrupt+0xb/0xc0 arch/x86/kernel/apic/apic.c:1100
softirqs last  enabled at (2696): [<ffffffff81462a6e>] invoke_softirq kernel/softirq.c:432 [inline]
softirqs last  enabled at (2696): [<ffffffff81462a6e>] __irq_exit_rcu+0x16e/0x1c0 kernel/softirq.c:636
softirqs last disabled at (2099): [<ffffffff81462a6e>] invoke_softirq kernel/softirq.c:432 [inline]
softirqs last disabled at (2099): [<ffffffff81462a6e>] __irq_exit_rcu+0x16e/0x1c0 kernel/softirq.c:636
CPU: 0 PID: 8673 Comm: syz-executor433 Not tainted 5.14.0-rc1-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0x452/0xc20 kernel/smp.c:967
Code: 0b 00 85 ed 74 4d 48 b8 00 00 00 00 00 fc ff df 4d 89 f4 4c 89 f5 49 c1 ec 03 83 e5 07 49 01 c4 83 c5 03 e8 40 48 0b 00 f3 90 <41> 0f b6 04 24 40 38 c5 7c 08 84 c0 0f 85 33 06 00 00 8b 43 08 31
RSP: 0018:ffffc90001bdfb20 EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff8880b9d570c0 RCX: 0000000000000000
RDX: ffff8880386ab880 RSI: ffffffff816a4130 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff816a4156 R11: 0000000000000000 R12: ffffed10173aae19
R13: 0000000000000001 R14: ffff8880b9d570c8 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000004bd370 CR3: 0000000035e11000 CR4: 00000000001506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 on_each_cpu_cond_mask+0x56/0xa0 kernel/smp.c:1133
 __flush_tlb_multi arch/x86/include/asm/paravirt.h:87 [inline]
 flush_tlb_multi arch/x86/mm/tlb.c:868 [inline]
 flush_tlb_mm_range+0x1d8/0x230 arch/x86/mm/tlb.c:954
 tlb_flush arch/x86/include/asm/tlb.h:23 [inline]
 tlb_flush_mmu_tlbonly include/asm-generic/tlb.h:426 [inline]
 tlb_flush_mmu_tlbonly include/asm-generic/tlb.h:416 [inline]
 tlb_flush_mmu mm/mmu_gather.c:248 [inline]
 tlb_finish_mmu+0x3e7/0x8c0 mm/mmu_gather.c:340
 exit_mmap+0x1ea/0x620 mm/mmap.c:3203
 __mmput+0x122/0x470 kernel/fork.c:1101
 mmput+0x58/0x60 kernel/fork.c:1122
 exit_mm kernel/exit.c:501 [inline]
 do_exit+0xae2/0x2a60 kernel/exit.c:812
 do_group_exit+0x125/0x310 kernel/exit.c:922
 __do_sys_exit_group kernel/exit.c:933 [inline]
 __se_sys_exit_group kernel/exit.c:931 [inline]
 __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:931
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x4427b9
Code: Unable to access opcode bytes at RIP 0x44278f.
RSP: 002b:00007ffe4e3bd608 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00000000004ba3d0 RCX: 00000000004427b9
RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 000000000000000d
R10: 000000000000000d R11: 0000000000000246 R12: 00000000004ba3d0
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001

Crashes (475):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	VM info	Title
ci-upstream-net-kasan-gce	2021/07/17 16:50	net-next	ab0441b4a920	f115ae98	.config	log	report	syz	C		BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/06/09 10:47	upstream	4c8684fe555e	5c2fe346	.config	log	report	syz	C		INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root	2020/08/16 01:04	upstream	c9c9735c46f5	424dd8e7	.config	log	report	syz	C
ci-upstream-kasan-gce-smack-root	2020/07/22 18:32	upstream	4fa640dc5230	128cd85f	.config	log	report	syz	C
ci-upstream-kasan-gce-selinux-root	2020/07/15 07:21	upstream	e9919e11e219	609fb517	.config	log	report	syz	C
ci-upstream-kasan-gce-root	2020/07/13 08:57	upstream	4437dd6e8f71	9ebcc5b1	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/12 11:25	linux-next	bc09acc9f224	bb3e5fe6	.config	log	report	syz	C
ci-upstream-kasan-gce	2021/10/17 05:46	upstream	304040fb4909	0c5d9412	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/10/16 22:24	upstream	304040fb4909	0c5d9412	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/10/15 10:18	upstream	ec681c53f8d2	0c5d9412	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/10/14 17:34	upstream	26d657410983	5462d470	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/10/13 22:38	upstream	348949d9a444	5462d470	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/10/12 07:14	upstream	fa5878760579	838e7e2c	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/10/12 05:54	upstream	fa5878760579	838e7e2c	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/10/11 09:45	upstream	64570fbc14f8	838e7e2c	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/10/11 01:13	upstream	efb52a7d9511	838e7e2c	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/10/10 19:21	upstream	efb52a7d9511	838e7e2c	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/10/10 10:03	upstream	7fd2bf83d59a	838e7e2c	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/10/09 21:13	upstream	5d6ab0bb408f	838e7e2c	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/10/08 10:29	upstream	1da38549dd64	efe0f24d	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/10/08 09:23	upstream	1da38549dd64	efe0f24d	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/10/07 22:51	upstream	4a16df549d23	efe0f24d	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/10/07 20:28	upstream	4a16df549d23	efe0f24d	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/10/05 16:49	upstream	f6274b06e326	0a63fd36	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/10/04 22:05	upstream	84b3e42564ac	ce697b49	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/10/04 20:07	upstream	84b3e42564ac	ce697b49	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/10/04 09:33	upstream	9e1ff307c779	ce697b49	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/10/03 22:28	upstream	6761a0ae9895	db0f5787	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/10/03 16:17	upstream	7b66f4393ad4	db0f5787	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/10/02 02:03	upstream	24f67d82c43c	cc80db95	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/09/30 21:36	upstream	02d5e016800d	1d849ab4	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/09/28 00:45	upstream	0513e464f900	78494d16	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/09/26 21:23	upstream	996148ee05d0	78494d16	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/09/24 22:43	upstream	4c4f0c2bf341	8cac236e	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/09/24 08:59	upstream	f9e36107ec70	8cac236e	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/09/24 04:14	upstream	f9e36107ec70	8cac236e	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/09/17 04:06	upstream	bdb575f87217	5b989942	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/09/16 16:34	upstream	ff1ffd71d5f0	aae492f2	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce	2021/09/16 13:02	upstream	b7213ffa0e58	07e953c1	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/09/16 02:35	upstream	b7213ffa0e58	07e953c1	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/09/16 00:55	upstream	b7213ffa0e58	07e953c1	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/09/15 11:04	upstream	3ca706c189db	07e953c1	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/09/14 20:27	upstream	1619b69edce1	07e953c1	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/09/11 06:03	upstream	926de8c4326c	5ae8508a	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/09/09 21:38	upstream	a3fa7a101dcf	e2776ee4	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/09/09 17:36	upstream	a3fa7a101dcf	e2776ee4	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/09/09 05:27	upstream	2d338201d531	e2776ee4	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/09/07 23:25	upstream	a2b28235335f	064c9eb7	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce	2021/09/07 02:47	upstream	27151f177827	6ca60148	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-qemu2-arm32	2021/08/11 20:37	upstream	bf152b0b41dc	6972b106	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-386	2021/08/04 22:31	upstream	251a1524293d	b97d64c9	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-net-this-kasan-gce	2021/09/27 23:11	net	763716a55cb1	78494d16	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce	2021/10/07 07:03	net-next	bcb2293d8106	62ee0987	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce	2021/09/27 11:08	net-next	d59bdda85eb7	78494d16	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce	2021/09/23 04:42	net-next	428168f99517	8cac236e	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce	2021/09/22 23:03	net-next	428168f99517	8cac236e	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce	2021/09/15 03:16	net-next	55bd079a3cb6	07e953c1	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce	2021/09/05 10:31	net-next	29ce8f970107	d236a457	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/10/14 20:32	linux-next	8006b911c90a	7aa5fe41	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/09/13 14:50	linux-next	24a36d3171e4	3ce60af8	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/10/05 08:04	upstream	84b3e42564ac	ce697b49	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/01/17 04:44	upstream	0da0a8a0a0e1	65a7a854	.config	log	report			info
ci-upstream-kasan-gce-root	2020/07/12 23:01	upstream	4437dd6e8f71	9ebcc5b1	.config	log	report
ci-upstream-kasan-gce-smack-root	2020/07/10 22:54	upstream	a581387e415b	18d18b59	.config	log	report
ci-upstream-kasan-gce-selinux-root	2020/07/08 22:22	upstream	0bddd227f3dc	bc238812	.config	log	report
ci-upstream-bpf-kasan-gce	2021/09/03 00:40	bpf	57f780f1c433	15cea0a3	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-bpf-next-kasan-gce	2020/11/02 15:12	bpf-next	cb5dc5b062a9	8bc4594f	.config	log	report			info
ci-upstream-linux-next-kasan-gce-root	2021/09/07 09:46	linux-next	b2bb710d34d5	6ca60148	.config	log	report			info	INFO: rcu detected stall in smp_call_function