BUG: soft lockup in smp_call

BUG: soft lockup in smp_call_function
Status: upstream: reported C repro on 2020/07/12 23:02
Reported-by: syzbot+cce3691658bef1b12ac9@syzkaller.appspotmail.com
First crash: 287d, last: 12h45m

Cause bisection: introduced by (bisect log) :
commit 5a781ccbd19e4664babcbe4b4ead7aa2b9283d22
Author: Vinicius Costa Gomes <vinicius.gomes@intel.com>
Date: Sat Sep 29 00:59:43 2018 +0000

tc: Add support for configuring the taprio scheduler

Crash: BUG: soft lockup in smp_call_function (log)
Repro: C syz .config

similar bugs (1):
Kernel	Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
upstream	INFO: rcu detected stall in smp_call_function	C			107	11d	266d	22/22	fixed on 2021/04/13 06:26

Sample crash report:

watchdog: BUG: soft lockup - CPU#0 stuck for 122s! [kworker/0:0:6866]
Modules linked in:
irq event stamp: 60796
hardirqs last  enabled at (60795): [<ffffffff88000c42>] asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:581
hardirqs last disabled at (60796): [<ffffffff87f1a99d>] irqentry_enter+0x1d/0x50 kernel/entry/common.c:318
softirqs last  enabled at (56852): [<ffffffff88000f2f>] asm_call_on_stack+0xf/0x20 arch/x86/entry/entry_64.S:706
softirqs last disabled at (56843): [<ffffffff88000f2f>] asm_call_on_stack+0xf/0x20 arch/x86/entry/entry_64.S:706
CPU: 0 PID: 6866 Comm: kworker/0:0 Not tainted 5.8.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events bpf_prog_free_deferred
RIP: 0010:csd_lock_wait kernel/smp.c:108 [inline]
RIP: 0010:smp_call_function_single+0x192/0x4f0 kernel/smp.c:382
Code: 10 8b 7c 24 1c 48 8d 74 24 40 48 89 44 24 50 48 8b 44 24 08 48 89 44 24 58 e8 fa f9 ff ff 41 89 c5 eb 07 e8 20 0a 0b 00 f3 90 <44> 8b 64 24 48 31 ff 41 83 e4 01 44 89 e6 e8 8b 06 0b 00 45 85 e4
RSP: 0018:ffffc900053d7960 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 1ffff92000a7af30 RCX: ffffffff81694405
RDX: ffff8880a8c925c0 RSI: ffffffff816943f0 RDI: 0000000000000005
RBP: ffffc900053d7a28 R08: 0000000000000001 R09: ffff8880ae736dc7
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000040
FS:  0000000000000000(0000) GS:ffff8880ae600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000558bf27b1f38 CR3: 000000009d31d000 CR4: 00000000001506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 smp_call_function_many_cond+0x1a4/0x990 kernel/smp.c:518
 smp_call_function_many kernel/smp.c:577 [inline]
 smp_call_function kernel/smp.c:599 [inline]
 on_each_cpu+0x4a/0x240 kernel/smp.c:698
 __purge_vmap_area_lazy+0x11e/0x1b70 mm/vmalloc.c:1350
 _vm_unmap_aliases.part.0+0x3cc/0x4d0 mm/vmalloc.c:1768
 _vm_unmap_aliases mm/vmalloc.c:1742 [inline]
 vm_remove_mappings mm/vmalloc.c:2234 [inline]
 __vunmap+0x5cd/0xac0 mm/vmalloc.c:2261
 __vfree mm/vmalloc.c:2318 [inline]
 vfree+0x88/0x140 mm/vmalloc.c:2348
 bpf_jit_free+0x19d/0x2e0
 bpf_prog_free_deferred+0x4ce/0x650 kernel/bpf/core.c:2140
 process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
 worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
 kthread+0x3b5/0x4a0 kernel/kthread.c:292
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 6874 Comm: syz-executor860 Not tainted 5.8.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:770 [inline]
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:160 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x51/0xe0 kernel/locking/spinlock.c:191
Code: 34 b6 89 48 ba 00 00 00 00 00 fc ff df 48 c1 e8 03 80 3c 10 00 0f 85 80 00 00 00 48 83 3d be f8 c1 01 00 74 5f 48 89 df 57 9d <0f> 1f 44 00 00 e8 75 d2 85 f9 eb 34 e8 ce d8 85 f9 48 c7 c0 88 34
RSP: 0018:ffffc90000da8cb0 EFLAGS: 00000082
RAX: 1ffffffff136c691 RBX: 0000000000000082 RCX: ffffffff815b6850
RDX: dffffc0000000000 RSI: 0000000000000004 RDI: 0000000000000082
RBP: ffffffff8cbf5140 R08: 0000000000000000 R09: ffffffff8cbf5143
R10: fffffbfff197ea28 R11: 0000000000000000 R12: ffff8880a7cf0340
R13: dffffc0000000000 R14: 1ffff920001b519b R15: ffffffff89be7140
FS:  0000000001593880(0000) GS:ffff8880ae700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006d50a0 CR3: 00000000a915e000 CR4: 00000000001506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 debug_object_activate+0x287/0x3e0 lib/debugobjects.c:664
 debug_hrtimer_activate kernel/time/hrtimer.c:420 [inline]
 debug_activate kernel/time/hrtimer.c:480 [inline]
 enqueue_hrtimer+0x27/0x3f0 kernel/time/hrtimer.c:969
 __run_hrtimer kernel/time/hrtimer.c:1541 [inline]
 __hrtimer_run_queues+0xc1e/0xfc0 kernel/time/hrtimer.c:1588
 hrtimer_interrupt+0x32a/0x930 kernel/time/hrtimer.c:1650
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1080 [inline]
 __sysvec_apic_timer_interrupt+0x142/0x5e0 arch/x86/kernel/apic/apic.c:1097
 asm_call_on_stack+0xf/0x20 arch/x86/entry/entry_64.S:706
 </IRQ>
 __run_on_irqstack arch/x86/include/asm/irq_stack.h:22 [inline]
 run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:48 [inline]
 sysvec_apic_timer_interrupt+0xb2/0xf0 arch/x86/kernel/apic/apic.c:1091
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:581
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:770 [inline]
RIP: 0010:on_each_cpu+0x149/0x240 kernel/smp.c:701
Code: 00 fc ff df 48 c1 e8 03 80 3c 10 00 0f 85 e6 00 00 00 48 83 3d 1f da 4c 08 00 0f 84 af 00 00 00 e8 9c f3 0a 00 48 89 df 57 9d <0f> 1f 44 00 00 e8 8d f3 0a 00 bf 01 00 00 00 e8 73 66 e6 ff 31 ff
RSP: 0018:ffffc90001477b70 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 0000000000000293 RCX: 1ffffffff1563891
RDX: ffff8880891121c0 RSI: ffffffff81695a74 RDI: 0000000000000293
RBP: 0000000000000200 R08: 0000000000000001 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 00000d4ae7fccc04 R14: 003b9aca00000000 R15: 0000000000000000
 clock_was_set+0x18/0x20 kernel/time/hrtimer.c:876
 timekeeping_inject_offset+0x436/0x570 kernel/time/timekeeping.c:1308
 do_adjtimex+0x28f/0xa30 kernel/time/timekeeping.c:2335
 do_clock_adjtime kernel/time/posix-timers.c:1109 [inline]
 __do_sys_clock_adjtime+0x155/0x250 kernel/time/posix-timers.c:1121
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x443719
Code: 18 89 d0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 fb 0f fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007fff96ad7e38 EFLAGS: 00000246 ORIG_RAX: 0000000000000131
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000443719
RDX: 0000000000443719 RSI: 0000000020000300 RDI: 0000000000000000
RBP: 00007fff96ad7e50 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000010000 R11: 0000000000000246 R12: 00007fff96ad7e60
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000

Crashes (239):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	VM info	Title
ci-upstream-kasan-gce-root	2020/08/16 01:04	upstream	c9c9735c	424dd8e7	.config	log	report	syz	C
ci-upstream-kasan-gce-smack-root	2020/07/22 18:32	upstream	4fa640dc	128cd85f	.config	log	report	syz	C
ci-upstream-kasan-gce-selinux-root	2020/07/15 07:21	upstream	e9919e11	609fb517	.config	log	report	syz	C
ci-upstream-kasan-gce-root	2020/07/13 08:57	upstream	4437dd6e	9ebcc5b1	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/12 11:25	linux-next	bc09acc9	bb3e5fe6	.config	log	report	syz	C
ci-upstream-kasan-gce-root	2021/04/19 10:08	upstream	bf05bf16	50f523d7	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/04/14 02:40	upstream	eebe426d	a184b83e	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/04/03 21:14	upstream	57fbdb15	6a81331a	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/04/03 19:23	upstream	57fbdb15	6a81331a	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/04/02 16:20	upstream	1678e493	6a81331a	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/04/01 22:17	upstream	ffd9fb54	6a81331a	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/03/27 02:19	upstream	db24726b	a8529b82	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/03/01 20:45	upstream	fe07bfda	183afb6c	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/01/31 15:53	upstream	6642d600	fc9fd31e	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/01/28 20:34	upstream	e5ff2cb9	7df34f59	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/01/28 17:41	upstream	76c057c8	7df34f59	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/01/26 19:24	upstream	13391c60	55a7d4df	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/01/22 20:26	upstream	83d09ad4	4080af96	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/01/19 03:42	upstream	1e2a199f	63631df1	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/01/18 16:25	upstream	19c329f6	63631df1	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/01/18 04:38	upstream	a1339d63	fd103621	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root	2021/01/17 20:19	upstream	0da0a8a0	813be542	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-net-this-kasan-gce	2021/01/28 03:13	net	b491e6a7	eefc07f2	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/04/21 11:10	linux-next	1216f02e	95777977	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/03/31 18:18	linux-next	93129492	6a81331a	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/02/04 14:14	linux-next	0e2c50f4	42b90a7c	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/01/23 09:57	linux-next	bc085f8f	52e37319	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/01/21 09:21	linux-next	bc085f8f	d4f4eca5	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/01/20 10:07	linux-next	647060f3	d4f4eca5	.config	log	report			info	BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/04/15 00:15	upstream	7f75285c	fcdb12ba	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-selinux-root	2021/04/13 14:37	upstream	89698bec	bfeda1b1	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root	2021/01/17 04:44	upstream	0da0a8a0	65a7a854	.config	log	report			info
ci-upstream-kasan-gce-root	2021/01/15 17:45	upstream	5ee88057	65a7a854	.config	log	report			info
ci-upstream-kasan-gce-selinux-root	2021/01/14 15:59	upstream	65f0d241	65a7a854	.config	log	report			info
ci-upstream-kasan-gce	2021/01/10 02:05	upstream	996e435f	2c1f2513	.config	log	report			info
ci-upstream-kasan-gce-root	2020/12/13 21:54	upstream	6bff9bb8	b22a7ec3	.config	log	report			info
ci-upstream-kasan-gce-smack-root	2020/12/11 15:15	upstream	33dc9614	ba24ffcd	.config	log	report			info
ci-upstream-kasan-gce-smack-root	2020/12/06 22:55	upstream	7059c2c0	c521566d	.config	log	report			info
ci-upstream-kasan-gce-root	2020/12/03 10:54	upstream	34816d20	e6b0d314	.config	log	report			info
ci-upstream-kasan-gce-smack-root	2020/11/30 13:45	upstream	b6505459	78d50c1d	.config	log	report			info
ci-upstream-kasan-gce-root	2020/11/27 21:46	upstream	99c710c4	486f93ef	.config	log	report			info
ci-upstream-kasan-gce-smack-root	2020/11/26 01:04	upstream	fa02fcd9	2f1cec62	.config	log	report			info
ci-upstream-kasan-gce-smack-root	2020/11/20 10:43	upstream	4d02da97	0767f13f	.config	log	report			info
ci-upstream-kasan-gce-root	2020/11/15 17:50	upstream	e28c0d7c	1bf9a662	.config	log	report			info
ci-upstream-kasan-gce-selinux-root	2020/11/13 16:22	upstream	585e5b17	e1140d25	.config	log	report			info
ci-upstream-kasan-gce-root	2020/11/12 21:42	upstream	af5043c8	77a55c8e	.config	log	report			info
ci-upstream-kasan-gce-selinux-root	2020/11/12 10:58	upstream	3d5e28bf	77a55c8e	.config	log	report			info
ci-upstream-kasan-gce-root	2020/11/07 14:10	upstream	659caaf6	64069d48	.config	log	report			info
ci-upstream-kasan-gce-selinux-root	2020/10/06 12:16	upstream	7575fdda	1880b4a9	.config	log	report			info
ci-upstream-kasan-gce-smack-root	2020/10/06 02:25	upstream	7575fdda	1880b4a9	.config	log	report			info
ci-upstream-kasan-gce-selinux-root	2020/10/05 17:42	upstream	549738f1	1880b4a9	.config	log	report			info
ci-upstream-kasan-gce-selinux-root	2020/10/04 16:30	upstream	22fbc037	5ef9c291	.config	log	report			info
ci-upstream-kasan-gce-smack-root	2020/10/04 11:32	upstream	22fbc037	5ef9c291	.config	log	report			info
ci-upstream-kasan-gce-selinux-root	2020/10/02 04:48	upstream	fcadab74	9602ddf4	.config	log	report			info
ci-upstream-kasan-gce-root	2020/07/12 23:01	upstream	4437dd6e	9ebcc5b1	.config	log	report
ci-upstream-kasan-gce-smack-root	2020/07/10 22:54	upstream	a581387e	18d18b59	.config	log	report
ci-upstream-kasan-gce-selinux-root	2020/07/08 22:22	upstream	0bddd227	bc238812	.config	log	report
ci-upstream-kasan-gce-386	2020/09/05 21:37	upstream	9322c47b	abf9ba4f	.config	log	report
ci-upstream-net-this-kasan-gce	2020/10/11 17:35	net	874fb9e2	4a77ae0b	.config	log	report			info
ci-upstream-bpf-next-kasan-gce	2020/11/02 15:12	bpf-next	cb5dc5b0	8bc4594f	.config	log	report			info
ci-upstream-net-kasan-gce	2020/09/25 21:38	net-next	aafe8853	4a006f63	.config	log	report			info
ci-upstream-linux-next-kasan-gce-root	2021/04/15 20:53	linux-next	1216f02e	c59079a6	.config	log	report			info	INFO: rcu detected stall in smp_call_function
ci-upstream-linux-next-kasan-gce-root	2021/01/07 11:18	linux-next	2d3811a4	c104d4a3	.config	log	report			info
ci-upstream-linux-next-kasan-gce-root	2020/12/06 20:33	linux-next	0eedceaf	c521566d	.config	log	report			info
ci-upstream-linux-next-kasan-gce-root	2020/11/27 19:18	linux-next	6174f052	486f93ef	.config	log	report			info
ci-upstream-linux-next-kasan-gce-root	2020/10/29 19:43	linux-next	3f267ec6	a0c7169a	.config	log	report			info
ci-upstream-linux-next-kasan-gce-root	2020/10/02 17:10	linux-next	2172e358	4969d6ca	.config	log	report			info