syzbot


BUG: soft lockup in smp_call_function
Status: upstream: reported C repro on 2020/07/12 23:02
Reported-by: syzbot+cce3691658bef1b12ac9@syzkaller.appspotmail.com
First crash: 687d, last: 2d14h

Cause bisection: introduced by (bisect log) :
commit 5a781ccbd19e4664babcbe4b4ead7aa2b9283d22
Author: Vinicius Costa Gomes <vinicius.gomes@intel.com>
Date: Sat Sep 29 00:59:43 2018 +0000

  tc: Add support for configuring the taprio scheduler

Crash: BUG: soft lockup in smp_call_function (log)
Repro: C syz .config
similar bugs (4):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 BUG: soft lockup in smp_call_function C error 1 33d 33d 0/1 upstream: reported C repro on 2022/04/23 05:26
android-54 BUG: soft lockup in smp_call_function 1 202d 202d 0/2 auto-closed as invalid on 2022/03/05 18:04
linux-4.14 BUG: soft lockup in smp_call_function C 1 3d18h 33d 0/1 upstream: reported C repro on 2022/04/23 04:43
upstream INFO: rcu detected stall in smp_call_function C 107 411d 666d 22/22 fixed on 2021/04/13 06:26

Sample crash report:
watchdog: BUG: soft lockup - CPU#1 stuck for 143s! [kworker/u4:1:3632]
Modules linked in:
irq event stamp: 45854
hardirqs last  enabled at (45853): [<ffffffff89600c02>] asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
hardirqs last disabled at (45854): [<ffffffff8945558b>] sysvec_apic_timer_interrupt+0xb/0xc0 arch/x86/kernel/apic/apic.c:1097
softirqs last  enabled at (17476): [<ffffffff84d03feb>] rcu_read_unlock_bh include/linux/rcupdate.h:754 [inline]
softirqs last  enabled at (17476): [<ffffffff84d03feb>] mod_peer_timer drivers/net/wireguard/timers.c:38 [inline]
softirqs last  enabled at (17476): [<ffffffff84d03feb>] wg_timers_handshake_initiated+0x1ab/0x300 drivers/net/wireguard/timers.c:184
softirqs last disabled at (17472): [<ffffffff84d03e88>] wg_timers_handshake_initiated+0x48/0x300 drivers/net/wireguard/timers.c:185
CPU: 1 PID: 3632 Comm: kworker/u4:1 Not tainted 5.16.0-rc6-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0x45c/0xc20 kernel/smp.c:969
Code: 00 00 00 fc ff df 4d 89 f4 4c 89 f5 49 c1 ec 03 83 e5 07 49 01 c4 83 c5 03 e8 40 78 0b 00 f3 90 41 0f b6 04 24 40 38 c5 7c 08 <84> c0 0f 85 33 06 00 00 8b 43 08 31 ff 83 e0 01 41 89 c7 89 c6 e8
RSP: 0018:ffffc90001acf9f8 EFLAGS: 00000206
RAX: 0000000000000000 RBX: ffff8880b9c41d40 RCX: 0000000000000000
RDX: ffff8880184a9d00 RSI: ffffffff816c2a00 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000001
R10: ffffffff816c2a26 R11: 0000000000000000 R12: ffffed10173883a9
R13: 0000000000000000 R14: ffff8880b9c41d48 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8880b9d00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffce47bb080 CR3: 000000000b88e000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 on_each_cpu_cond_mask+0x56/0xa0 kernel/smp.c:1135
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:1112 [inline]
 text_poke_bp_batch+0x1b3/0x560 arch/x86/kernel/alternative.c:1297
 text_poke_flush arch/x86/kernel/alternative.c:1451 [inline]
 text_poke_flush arch/x86/kernel/alternative.c:1448 [inline]
 text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1458
 arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
 jump_label_update+0x1d5/0x430 kernel/jump_label.c:830
 static_key_enable_cpuslocked+0x1b1/0x260 kernel/jump_label.c:177
 static_key_enable+0x16/0x20 kernel/jump_label.c:190
 toggle_allocation_gate mm/kfence/core.c:732 [inline]
 toggle_allocation_gate+0x100/0x390 mm/kfence/core.c:724
 process_one_work+0x9b2/0x1690 kernel/workqueue.c:2298
 worker_thread+0x658/0x11f0 kernel/workqueue.c:2445
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
----------------
Code disassembly (best guess), 5 bytes skipped:
   0:	df 4d 89             	fisttps -0x77(%rbp)
   3:	f4                   	hlt
   4:	4c 89 f5             	mov    %r14,%rbp
   7:	49 c1 ec 03          	shr    $0x3,%r12
   b:	83 e5 07             	and    $0x7,%ebp
   e:	49 01 c4             	add    %rax,%r12
  11:	83 c5 03             	add    $0x3,%ebp
  14:	e8 40 78 0b 00       	callq  0xb7859
  19:	f3 90                	pause
  1b:	41 0f b6 04 24       	movzbl (%r12),%eax
  20:	40 38 c5             	cmp    %al,%bpl
  23:	7c 08                	jl     0x2d
* 25:	84 c0                	test   %al,%al <-- trapping instruction
  27:	0f 85 33 06 00 00    	jne    0x660
  2d:	8b 43 08             	mov    0x8(%rbx),%eax
  30:	31 ff                	xor    %edi,%edi
  32:	83 e0 01             	and    $0x1,%eax
  35:	41 89 c7             	mov    %eax,%r15d
  38:	89 c6                	mov    %eax,%esi
  3a:	e8                   	.byte 0xe8

Fix bisection attempts:
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-upstream-net-this-kasan-gce 2022/05/24 09:14 net 7fb0269720d7 6b3c5e64 .config log report syz C
Crashes (759):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-upstream-net-this-kasan-gce 2021/12/28 04:31 net 79b69a83705e 6b3c5e64 .config log report syz C BUG: soft lockup in smp_call_function
ci-upstream-net-this-kasan-gce 2021/10/22 01:10 net e0bfcf9c77d9 c5cb7da8 .config log report syz C BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2021/10/21 20:33 net-next dfcb63ce1de6 c5cb7da8 .config log report syz C BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2021/07/17 16:50 net-next ab0441b4a920 f115ae98 .config log report syz C BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/04/16 15:03 upstream 59250f8a7f3a 8bcc32a6 .config log report syz C INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2021/06/09 10:47 upstream 4c8684fe555e 5c2fe346 .config log report syz C INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2020/08/16 01:04 upstream c9c9735c46f5 424dd8e7 .config log report syz C
ci-upstream-kasan-gce-smack-root 2020/07/22 18:32 upstream 4fa640dc5230 128cd85f .config log report syz C
ci-upstream-kasan-gce-selinux-root 2020/07/15 07:21 upstream e9919e11e219 609fb517 .config log report syz C
ci-upstream-kasan-gce-root 2020/07/13 08:57 upstream 4437dd6e8f71 9ebcc5b1 .config log report syz C
ci-upstream-linux-next-kasan-gce-root 2020/08/12 11:25 linux-next bc09acc9f224 bb3e5fe6 .config log report syz C
ci-upstream-kasan-gce-root 2022/04/24 08:49 upstream 22da5264abf4 131df97d .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/03/22 04:28 upstream eaa54b1458ca e2d91b1d .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/03/19 22:40 upstream 97e9c8eb4bb1 e2d91b1d .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/03/13 21:40 upstream f0e18b03fcaf 9e8eaa75 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/03/09 10:01 upstream 330f4c53d3c2 9e8eaa75 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/03/08 09:00 upstream ea4424be1688 7bdd8b2c .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/03/05 14:24 upstream ac84e82f78cb 45a13a73 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/03/02 11:52 upstream fb184c4af9b9 45a13a73 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce 2022/03/01 15:10 upstream 719fce7539cd 45a13a73 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/25 16:02 upstream 53ab78cd6d5a 7c337266 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/02/24 22:45 upstream 73878e5eb1bd b28851a4 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/22 22:12 upstream 917bbdb107f8 6e821dbf .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/02/19 10:29 upstream 4f12b742eb2b 3cd800e4 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/02/17 15:51 upstream f71077a4d84b 3cd800e4 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/02/12 12:20 upstream 83e396641110 8b9ca619 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/11 22:16 upstream 1d41d2e82623 8b9ca619 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/10 17:07 upstream e3c85076d7a6 0b33604d .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/07 12:19 upstream dfd42facf1e4 a7dab638 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/02/05 23:57 upstream 90c9e950c0de a7dab638 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/02/05 01:21 upstream dcb85f85fa6f e13a05ed .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/02/04 15:08 upstream dcb85f85fa6f a3e470b2 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/02 23:58 upstream 27bb0b18c208 4ebb2798 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/02/01 15:07 upstream 26291c54e111 c1c1631d .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/02/01 02:09 upstream 26291c54e111 6b7c57fe .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/01 00:51 upstream 26291c54e111 6b7c57fe .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/01/31 10:20 upstream 26291c54e111 6b7c57fe .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/01/31 06:00 upstream 26291c54e111 495e00c5 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/01/30 04:20 upstream f8c7e4ede46f 495e00c5 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/01/25 11:14 upstream a08b41ab9e2e 2cbffd88 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/01/24 20:52 upstream dd81e1c7d5fb 2cbffd88 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/01/23 20:15 upstream dd81e1c7d5fb 214351e1 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/01/22 16:15 upstream 0809edbae347 214351e1 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/01/21 03:59 upstream 2c271fe77d52 ab3d9f17 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/01/20 17:30 upstream fa2e1ba3e9e3 b838eb76 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/01/19 21:56 upstream 1d1df41c5a33 5da9499f .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/01/18 16:15 upstream e3a8b6a1e70c 731a2d23 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-386 2021/11/14 13:28 upstream c8c109546a19 83f5c9b5 .config log report info BUG: soft lockup in smp_call_function
ci-qemu2-arm32 2021/08/11 20:37 upstream bf152b0b41dc 6972b106 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-net-this-kasan-gce 2022/01/19 12:43 net 99845220d3c3 0620189b .config log report info BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2022/02/15 08:13 net-next 5a8fb33e5305 8b9ca619 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2022/01/20 22:09 net-next fe8152b38d3a b838eb76 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2022/01/19 20:55 net-next fe8152b38d3a 5da9499f .config log report info BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2022/01/19 18:57 net-next fe8152b38d3a 0620189b .config log report info BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2022/01/19 07:41 net-next fe8152b38d3a 731a2d23 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2022/01/19 04:20 net-next fe8152b38d3a 731a2d23 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2022/01/18 19:22 net-next fe8152b38d3a 731a2d23 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2022/01/18 11:25 net-next fe8152b38d3a 731a2d23 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root 2022/01/05 12:40 linux-next 7a769a3922d8 0a2584dd .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/04/13 20:00 upstream a19944809fe9 b17b2923 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/04/13 02:29 upstream a19944809fe9 dacb3f1c .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/03/28 13:34 upstream ae085d7f9365 ee339263 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/02/26 09:40 upstream 9137eda53752 45a13a73 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/01/20 00:44 upstream 1d1df41c5a33 5da9499f .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2021/01/17 04:44 upstream 0da0a8a0a0e1 65a7a854 .config log report info
ci-upstream-kasan-gce-root 2020/07/12 23:01 upstream 4437dd6e8f71 9ebcc5b1 .config log report
ci-upstream-kasan-gce-smack-root 2020/07/10 22:54 upstream a581387e415b 18d18b59 .config log report
ci-upstream-kasan-gce-selinux-root 2020/07/08 22:22 upstream 0bddd227f3dc bc238812 .config log report
ci-upstream-bpf-kasan-gce 2021/09/03 00:40 bpf 57f780f1c433 15cea0a3 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-net-kasan-gce 2022/01/19 01:56 net-next fe8152b38d3a 731a2d23 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-net-kasan-gce 2022/01/18 05:09 net-next fe8152b38d3a 731a2d23 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-net-kasan-gce 2022/01/17 22:23 net-next fe8152b38d3a 731a2d23 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-bpf-next-kasan-gce 2020/11/02 15:12 bpf-next cb5dc5b062a9 8bc4594f .config log report info