syzbot

 sign-in | mailing list | source | docs
🐞 Open [954] 🐞 Fixed [4037] 🐞 Invalid [8979] 📈 Kernel Health 📈 Bug Lifetimes 📈 Fuzzing 📈 Crashes

BUG: soft lockup in smp_call_function

Status: upstream: reported C repro on 2020/07/12 23:02
Reported-by: syzbot+cce3691658bef1b12ac9@syzkaller.appspotmail.com
First crash: 816d, last: 1h19m

Cause bisection: introduced by (bisect log) :
commit 5a781ccbd19e4664babcbe4b4ead7aa2b9283d22
Author: Vinicius Costa Gomes <vinicius.gomes@intel.com>
Date: Sat Sep 29 00:59:43 2018 +0000

  tc: Add support for configuring the taprio scheduler

Crash: BUG: soft lockup in smp_call_function (log)
Repro: C syz .config
similar bugs (4):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 BUG: soft lockup in smp_call_function C error 2 34d 163d 0/1 upstream: reported C repro on 2022/04/23 05:26
android-54 BUG: soft lockup in smp_call_function 1 332d 332d 0/2 auto-closed as invalid on 2022/03/05 18:04
linux-4.14 BUG: soft lockup in smp_call_function C 1 13d 163d 0/1 upstream: reported C repro on 2022/04/23 04:43
upstream INFO: rcu detected stall in smp_call_function C 107 541d 796d 22/24 fixed on 2021/04/13 06:26

Sample crash report:
hrtimer: interrupt took 626456594 ns
watchdog: BUG: soft lockup - CPU#1 stuck for 160s! [kworker/u4:0:3654]
Modules linked in:
irq event stamp: 3342
hardirqs last  enabled at (3341): [<ffffffff81716247>] smp_call_function_many_cond+0x10e7/0x1430 kernel/smp.c:980
hardirqs last disabled at (3342): [<ffffffff897ec0bb>] sysvec_apic_timer_interrupt+0xb/0xc0 arch/x86/kernel/apic/apic.c:1106
softirqs last  enabled at (3310): [<ffffffff89185805>] spin_unlock_bh include/linux/spinlock.h:394 [inline]
softirqs last  enabled at (3310): [<ffffffff89185805>] batadv_nc_purge_paths+0x2a5/0x3a0 net/batman-adv/network-coding.c:471
softirqs last disabled at (3308): [<ffffffff8918563f>] spin_lock_bh include/linux/spinlock.h:354 [inline]
softirqs last disabled at (3308): [<ffffffff8918563f>] batadv_nc_purge_paths+0xdf/0x3a0 net/batman-adv/network-coding.c:442
CPU: 1 PID: 3654 Comm: kworker/u4:0 Not tainted 5.19.0-syzkaller-13930-g7ebfc85e2cd7 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/22/2022
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:414 [inline]
RIP: 0010:smp_call_function_many_cond+0x5c3/0x1430 kernel/smp.c:988
Code: 89 ee e8 d0 ac 0a 00 85 ed 74 48 48 8b 44 24 08 49 89 c4 83 e0 07 49 c1 ec 03 48 89 c5 4d 01 f4 83 c5 03 e8 ef af 0a 00 f3 90 <41> 0f b6 04 24 40 38 c5 7c 08 84 c0 0f 85 b5 0b 00 00 8b 43 08 31
RSP: 0018:ffffc90003a7f948 EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff8880b9a42580 RCX: 0000000000000000
RDX: ffff88801e355880 RSI: ffffffff81715721 RDI: 0000000000000005
RBP: 0000000000000003 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffffed10173484b1
R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000200019d4 CR3: 000000000bc8e000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 on_each_cpu_cond_mask+0x56/0xa0 kernel/smp.c:1154
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:1302 [inline]
 text_poke_bp_batch+0x233/0x6c0 arch/x86/kernel/alternative.c:1490
 text_poke_flush arch/x86/kernel/alternative.c:1660 [inline]
 text_poke_flush arch/x86/kernel/alternative.c:1657 [inline]
 text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1667
 arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
 jump_label_update+0x32f/0x410 kernel/jump_label.c:801
 static_key_enable_cpuslocked+0x1b1/0x260 kernel/jump_label.c:177
 static_key_enable+0x16/0x20 kernel/jump_label.c:190
 toggle_allocation_gate mm/kfence/core.c:811 [inline]
 toggle_allocation_gate+0x100/0x390 mm/kfence/core.c:803
 process_one_work+0x991/0x1610 kernel/workqueue.c:2289
 worker_thread+0x665/0x1080 kernel/workqueue.c:2436
 kthread+0x2e4/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
 </TASK>

Fix bisection attempts:
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-upstream-net-this-kasan-gce 2022/05/24 09:14 net 7fb0269720d7 6b3c5e64 .config log report syz C
* Struck through repros no longer work on HEAD.
Crashes (790):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-upstream-kasan-gce 2022/08/15 21:07 upstream 7ebfc85e2cd7 8dfcaa3d .config log report syz C BUG: soft lockup in smp_call_function
ci-upstream-net-this-kasan-gce 2021/12/28 04:31 net 79b69a83705e 6b3c5e64 .config log report syz C BUG: soft lockup in smp_call_function
ci-upstream-net-this-kasan-gce 2021/10/22 01:10 net e0bfcf9c77d9 c5cb7da8 .config log report syz C BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2021/10/21 20:33 net-next dfcb63ce1de6 c5cb7da8 .config log report syz C BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2021/07/17 16:50 net-next ab0441b4a920 f115ae98 .config log report syz C BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root 2022/10/03 04:13 linux-next aaa11ce2ffc8 feb56351 .config log report syz C BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root 2022/08/26 02:07 linux-next b5d939c95186 9b5bf4cd .config log report syz C BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/04/16 15:03 upstream 59250f8a7f3a 8bcc32a6 .config log report syz C INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2021/06/09 10:47 upstream 4c8684fe555e 5c2fe346 .config log report syz C INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2020/08/16 01:04 upstream c9c9735c46f5 424dd8e7 .config log report syz C
ci-upstream-kasan-gce-smack-root 2020/07/22 18:32 upstream 4fa640dc5230 128cd85f .config log report syz C
ci-upstream-kasan-gce-selinux-root 2020/07/15 07:21 upstream e9919e11e219 609fb517 .config log report syz C
ci-upstream-kasan-gce-root 2020/07/13 08:57 upstream 4437dd6e8f71 9ebcc5b1 .config log report syz C
ci-upstream-linux-next-kasan-gce-root 2020/08/12 11:25 linux-next bc09acc9f224 bb3e5fe6 .config log report syz C
ci-upstream-kasan-gce-root 2022/08/25 21:35 upstream 3f5c20055a64 9b5bf4cd .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/04/24 08:49 upstream 22da5264abf4 131df97d .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/03/22 04:28 upstream eaa54b1458ca e2d91b1d .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/03/19 22:40 upstream 97e9c8eb4bb1 e2d91b1d .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/03/13 21:40 upstream f0e18b03fcaf 9e8eaa75 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/03/09 10:01 upstream 330f4c53d3c2 9e8eaa75 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/03/08 09:00 upstream ea4424be1688 7bdd8b2c .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/03/05 14:24 upstream ac84e82f78cb 45a13a73 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/03/02 11:52 upstream fb184c4af9b9 45a13a73 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce 2022/03/01 15:10 upstream 719fce7539cd 45a13a73 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/25 16:02 upstream 53ab78cd6d5a 7c337266 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/02/24 22:45 upstream 73878e5eb1bd b28851a4 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/22 22:12 upstream 917bbdb107f8 6e821dbf .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/02/19 10:29 upstream 4f12b742eb2b 3cd800e4 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/02/17 15:51 upstream f71077a4d84b 3cd800e4 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/02/12 12:20 upstream 83e396641110 8b9ca619 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/11 22:16 upstream 1d41d2e82623 8b9ca619 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/10 17:07 upstream e3c85076d7a6 0b33604d .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/07 12:19 upstream dfd42facf1e4 a7dab638 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/02/05 23:57 upstream 90c9e950c0de a7dab638 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/02/05 01:21 upstream dcb85f85fa6f e13a05ed .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/02/04 15:08 upstream dcb85f85fa6f a3e470b2 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/02/02 23:58 upstream 27bb0b18c208 4ebb2798 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-386 2021/11/14 13:28 upstream c8c109546a19 83f5c9b5 .config log report info BUG: soft lockup in smp_call_function
ci-qemu2-arm32 2021/08/11 20:37 upstream bf152b0b41dc 6972b106 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-net-this-kasan-gce 2022/01/19 12:43 net 99845220d3c3 0620189b .config log report info BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2022/02/15 08:13 net-next 5a8fb33e5305 8b9ca619 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root 2022/08/23 06:12 linux-next 05477f3653b8 26a13b38 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/10/03 19:41 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci bbed346d5a96 feb56351 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/27 23:41 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci c194837ebb57 75c78242 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/23 23:13 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci c194837ebb57 0042f2b4 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/21 04:37 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 5aa266bb455b c4b8ccfd .config log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/20 08:47 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a6b443748715 7c41a9ba .config log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/19 11:14 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a6b443748715 dd9a85ff .config log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/19 07:22 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a6b443748715 dd9a85ff .config log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/10 22:15 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a6b443748715 356d8217 .config log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/05 22:57 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 85413d1e802e 9dcd38fc .config log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/04 22:20 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 85413d1e802e 28811d0a .config log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/09/25 15:57 upstream 105a36f3694e 0042f2b4 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/09/13 10:30 upstream e839a756012b a08652b0 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/09/12 13:54 upstream 80e78fcce86d 356d8217 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/09/12 11:10 upstream 80e78fcce86d 356d8217 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/09/02 05:18 upstream 42e66b1cc3a0 a805568e .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/08/30 16:01 upstream dcf8e5633e2e 4a380809 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/08/22 18:39 upstream 1c23f9e627a7 26a13b38 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/07/23 01:06 upstream 4ba1329cbb94 22343af4 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/07/22 04:48 upstream 68e77ffbfd06 5e6028b9 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/07/15 11:32 upstream e5d523f1ae8f 5d921b08 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/07/15 03:35 upstream 9bd572ec7a66 5d921b08 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/06/14 16:15 upstream b13baccc3850 127d1faf .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/06/09 15:53 upstream 6bfb56e93bce 0d5abf15 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/04/13 20:00 upstream a19944809fe9 b17b2923 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/04/13 02:29 upstream a19944809fe9 dacb3f1c .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/03/28 13:34 upstream ae085d7f9365 ee339263 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/02/26 09:40 upstream 9137eda53752 45a13a73 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2021/01/17 04:44 upstream 0da0a8a0a0e1 65a7a854 .config log report info
ci-upstream-kasan-gce-root 2020/07/12 23:01 upstream 4437dd6e8f71 9ebcc5b1 .config log report
ci-upstream-kasan-gce-smack-root 2020/07/10 22:54 upstream a581387e415b 18d18b59 .config log report
ci-upstream-kasan-gce-selinux-root 2020/07/08 22:22 upstream 0bddd227f3dc bc238812 .config log report
ci-upstream-bpf-kasan-gce 2021/09/03 00:40 bpf 57f780f1c433 15cea0a3 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-bpf-next-kasan-gce 2020/11/02 15:12 bpf-next cb5dc5b062a9 8bc4594f .config log report info
ci-upstream-linux-next-kasan-gce-root 2022/09/27 13:58 linux-next aaa11ce2ffc8 87840e00 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-linux-next-kasan-gce-root 2022/08/23 19:53 linux-next 05477f3653b8 cea8b0f7 .config log report info INFO: rcu detected stall in smp_call_function
ci-upstream-linux-next-kasan-gce-root 2022/06/15 15:28 linux-next 6012273897fe 1719ee24 .config log report info INFO: rcu detected stall in smp_call_function
* Struck through repros no longer work on HEAD.