BUG: soft lockup in smp_call

BUG: soft lockup in smp_call_function
Status: upstream: reported C repro on 2020/07/12 23:02
Reported-by: syzbot+cce3691658bef1b12ac9@syzkaller.appspotmail.com
First crash: 380d, last: 12h26m

Cause bisection: introduced by (bisect log) :
commit 5a781ccbd19e4664babcbe4b4ead7aa2b9283d22
Author: Vinicius Costa Gomes <vinicius.gomes@intel.com>
Date: Sat Sep 29 00:59:43 2018 +0000

tc: Add support for configuring the taprio scheduler

Crash: BUG: soft lockup in smp_call_function (log)
Repro: C syz .config

similar bugs (1):
Kernel	Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
upstream	INFO: rcu detected stall in smp_call_function	C			107	104d	359d	22/22	fixed on 2021/04/13 06:26

Sample crash report:

watchdog: BUG: soft lockup - CPU#0 stuck for 118s! [syz-executor433:8673]
Modules linked in:
irq event stamp: 28124
hardirqs last  enabled at (28123): [<ffffffff89400c02>] asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
hardirqs last disabled at (28124): [<ffffffff892a1efb>] sysvec_apic_timer_interrupt+0xb/0xc0 arch/x86/kernel/apic/apic.c:1100
softirqs last  enabled at (2696): [<ffffffff81462a6e>] invoke_softirq kernel/softirq.c:432 [inline]
softirqs last  enabled at (2696): [<ffffffff81462a6e>] __irq_exit_rcu+0x16e/0x1c0 kernel/softirq.c:636
softirqs last disabled at (2099): [<ffffffff81462a6e>] invoke_softirq kernel/softirq.c:432 [inline]
softirqs last disabled at (2099): [<ffffffff81462a6e>] __irq_exit_rcu+0x16e/0x1c0 kernel/softirq.c:636
CPU: 0 PID: 8673 Comm: syz-executor433 Not tainted 5.14.0-rc1-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0x452/0xc20 kernel/smp.c:967
Code: 0b 00 85 ed 74 4d 48 b8 00 00 00 00 00 fc ff df 4d 89 f4 4c 89 f5 49 c1 ec 03 83 e5 07 49 01 c4 83 c5 03 e8 40 48 0b 00 f3 90 <41> 0f b6 04 24 40 38 c5 7c 08 84 c0 0f 85 33 06 00 00 8b 43 08 31
RSP: 0018:ffffc90001bdfb20 EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff8880b9d570c0 RCX: 0000000000000000
RDX: ffff8880386ab880 RSI: ffffffff816a4130 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff816a4156 R11: 0000000000000000 R12: ffffed10173aae19
R13: 0000000000000001 R14: ffff8880b9d570c8 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000004bd370 CR3: 0000000035e11000 CR4: 00000000001506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 on_each_cpu_cond_mask+0x56/0xa0 kernel/smp.c:1133
 __flush_tlb_multi arch/x86/include/asm/paravirt.h:87 [inline]
 flush_tlb_multi arch/x86/mm/tlb.c:868 [inline]
 flush_tlb_mm_range+0x1d8/0x230 arch/x86/mm/tlb.c:954
 tlb_flush arch/x86/include/asm/tlb.h:23 [inline]
 tlb_flush_mmu_tlbonly include/asm-generic/tlb.h:426 [inline]
 tlb_flush_mmu_tlbonly include/asm-generic/tlb.h:416 [inline]
 tlb_flush_mmu mm/mmu_gather.c:248 [inline]
 tlb_finish_mmu+0x3e7/0x8c0 mm/mmu_gather.c:340
 exit_mmap+0x1ea/0x620 mm/mmap.c:3203
 __mmput+0x122/0x470 kernel/fork.c:1101
 mmput+0x58/0x60 kernel/fork.c:1122
 exit_mm kernel/exit.c:501 [inline]
 do_exit+0xae2/0x2a60 kernel/exit.c:812
 do_group_exit+0x125/0x310 kernel/exit.c:922
 __do_sys_exit_group kernel/exit.c:933 [inline]
 __se_sys_exit_group kernel/exit.c:931 [inline]
 __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:931
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x4427b9
Code: Unable to access opcode bytes at RIP 0x44278f.
RSP: 002b:00007ffe4e3bd608 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00000000004ba3d0 RCX: 00000000004427b9
RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 000000000000000d
R10: 000000000000000d R11: 0000000000000246 R12: 00000000004ba3d0
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001

Crashes (334):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	VM info	Title
ci-upstream-net-kasan-gce	2021/07/17 16:50	net-next	ab0441b4a920	f115ae98	.config	log	report	syz	C		BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/06/09 10:47	upstream	4c8684fe555e	5c2fe346	.config	log	report	syz	C		INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root	2020/08/16 01:04	upstream	c9c9735c46f5	424dd8e7	.config	log	report	syz	C
ci-upstream-kasan-gce-smack-root	2020/07/22 18:32	upstream	4fa640dc5230	128cd85f	.config	log	report	syz	C
ci-upstream-kasan-gce-selinux-root	2020/07/15 07:21	upstream	e9919e11e219	609fb517	.config	log	report	syz	C
ci-upstream-kasan-gce-root	2020/07/13 08:57	upstream	4437dd6e8f71	9ebcc5b1	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/12 11:25	linux-next	bc09acc9f224	bb3e5fe6	.config	log	report	syz	C
ci-upstream-kasan-gce-smack-root	2021/07/22 04:29	upstream	7b6ae471e541	29c3f20f	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/07/19 23:41	upstream	2734d6c1b1a0	bc48c9ab	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/07/17 13:32	upstream	d980cc0620ae	f115ae98	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/07/16 02:42	upstream	dd9c7df94c1b	f115ae98	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/07/15 06:30	upstream	8096acd7442e	b9a2f64e	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/07/14 17:24	upstream	40226a3d96ef	94e0b707	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce	2021/07/12 07:10	upstream	e73f0f0ee754	a4869c92	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce	2021/07/07 11:07	upstream	3dbdb38e2869	4846d5c1	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/07/07 08:19	upstream	3dbdb38e2869	4846d5c1	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/07/06 20:39	upstream	3dbdb38e2869	cca78469	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/07/06 13:06	upstream	3dbdb38e2869	6c4484eb	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/07/06 10:59	upstream	3dbdb38e2869	6c4484eb	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/07/03 05:13	upstream	3dbdb38e2869	55aa55c2	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/06/30 21:17	upstream	df04fbe8680b	38a885d1	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce	2021/06/25 05:00	upstream	4a09d388f2ab	0edbbe31	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/06/24 09:53	upstream	7426cedc7dad	ec865f6a	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/06/23 08:32	upstream	0c18f29aae7c	aba2b2fb	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/06/21 18:32	upstream	13311e74253f	aba2b2fb	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/06/12 16:52	upstream	ad347abe4a98	1ba81399	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/06/06 15:45	upstream	f5b6eb1e0182	500c2339	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/06/03 23:29	upstream	f88cd3fb9df2	0740de69	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/06/03 03:00	upstream	324c92e5e0ee	0740de69	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/06/02 05:56	upstream	231bc5390667	032639db	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/05/30 15:25	upstream	b90e90f40b4f	325a8dab	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/05/29 22:22	upstream	6799d4f2da49	325a8dab	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/05/25 22:41	upstream	a050a6d2b7e8	93d3a9f6	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/05/24 04:56	upstream	6ebb6814a1ef	3c7fef33	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-386	2021/07/10 02:18	upstream	3dbdb38e2869	8f5a7b8c	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-386	2021/07/06 09:40	upstream	3dbdb38e2869	6c4484eb	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-qemu2-arm32	2021/06/20 13:01	upstream	bf152b0b41dc	aba2b2fb	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce	2021/05/31 10:07	net-next	a729b8e6ec3d	325a8dab	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce	2021/05/27 03:38	net-next	c7a551b2e44a	858ea628	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/07/22 10:44	linux-next	7468cbf5c917	29c3f20f	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/07/14 11:36	linux-next	c0d438dbc0b7	484502bd	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/07/08 02:58	linux-next	ee268dee405b	95793bce	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/06/25 16:07	linux-next	a1f92694393a	ae6bf8dd	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/06/09 01:12	linux-next	a1f92694393a	5c2fe346	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/05/29 04:15	linux-next	a1f92694393a	858ea628	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/07/23 16:04	upstream	9bead1b58c4c	bc5f1d88	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/07/21 18:58	upstream	8cae8cd89f05	29c3f20f	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce	2021/07/19 16:51	upstream	2734d6c1b1a0	e6a17580	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root	2021/07/19 10:46	upstream	2734d6c1b1a0	e6a17580	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/07/18 15:27	upstream	1d67c8d993ba	f115ae98	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root	2021/07/02 14:03	upstream	3dbdb38e2869	658ebc66	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/06/23 21:27	upstream	7266f2030eb0	fe4ab389	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/06/22 20:54	upstream	0c18f29aae7c	aba2b2fb	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/06/16 19:10	upstream	6b00bc639f1f	c06f97ad	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root	2021/06/15 10:43	upstream	009c9aa5be65	58636922	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/06/14 11:48	upstream	009c9aa5be65	1ba81399	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/06/13 23:12	upstream	e4e453434a19	1ba81399	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/06/09 15:10	upstream	368094df48e6	84fe5d96	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/06/08 01:49	upstream	614124bea77e	e59537be	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/01/17 04:44	upstream	0da0a8a0a0e1	65a7a854	.config	log	report			info
ci-upstream-kasan-gce-root	2020/07/12 23:01	upstream	4437dd6e8f71	9ebcc5b1	.config	log	report
ci-upstream-kasan-gce-smack-root	2020/07/10 22:54	upstream	a581387e415b	18d18b59	.config	log	report
ci-upstream-kasan-gce-selinux-root	2020/07/08 22:22	upstream	0bddd227f3dc	bc238812	.config	log	report
ci-upstream-net-this-kasan-gce	2021/06/07 22:29	net	3822d0670c9d	e59537be	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-net-kasan-gce	2021/06/28 07:55	net-next	ff8744b5eb11	9d2ab5df	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-net-kasan-gce	2021/06/15 04:02	net-next	2d7ff2d83cac	1ba81399	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-net-kasan-gce	2021/05/31 02:08	net-next	d3f2c48de7b8	325a8dab	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-bpf-next-kasan-gce	2020/11/02 15:12	bpf-next	cb5dc5b062a9	8bc4594f	.config	log	report			info
ci-upstream-linux-next-kasan-gce-root	2021/07/08 09:37	linux-next	e2f74b13dbe6	95793bce	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/06/29 21:08	linux-next	a1f92694393a	a4fccb01	.config	log	report			info	INFO: rcu detected stall in smp_call_function