syzbot

 sign-in | mailing list | source | docs
🐞 Open [1163] 🐞 Fixed [4320] 🐞 Invalid [9666] 📈 Kernel Health 📈 Bug Lifetimes 📈 Fuzzing 📈 Crashes

BUG: soft lockup in smp_call_function

Status: upstream: reported C repro on 2020/07/12 23:02
Reported-by: syzbot+cce3691658bef1b12ac9@syzkaller.appspotmail.com
First crash: 942d, last: 18d

Cause bisection: introduced by (bisect log) :
commit 5a781ccbd19e4664babcbe4b4ead7aa2b9283d22
Author: Vinicius Costa Gomes <vinicius.gomes@intel.com>
Date: Sat Sep 29 00:59:43 2018 +0000

  tc: Add support for configuring the taprio scheduler

Crash: BUG: soft lockup in smp_call_function (log)
Repro: C syz .config
similar bugs (5):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 BUG: soft lockup in smp_call_function C error 2 159d 288d 0/1 upstream: reported C repro on 2022/04/23 05:26
android-54 BUG: soft lockup in smp_call_function 1 457d 457d 0/2 auto-closed as invalid on 2022/03/05 18:04
linux-4.14 BUG: soft lockup in smp_call_function C 1 16d 288d 0/1 upstream: reported C repro on 2022/04/23 04:43
android-5-15 BUG: soft lockup in smp_call_function 1 72d 72d 0/2 premoderation: reported on 2022/11/25 06:05
upstream INFO: rcu detected stall in smp_call_function C 107 666d 921d 22/24 fixed on 2021/04/13 06:26

Sample crash report:
watchdog: BUG: soft lockup - CPU#0 stuck for 143s! [syz-executor189:3665]
Modules linked in:
irq event stamp: 202716
hardirqs last  enabled at (202715): [<ffffffff8a00140a>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:649
hardirqs last disabled at (202716): [<ffffffff89ecf9cf>] sysvec_apic_timer_interrupt+0xf/0xc0 arch/x86/kernel/apic/apic.c:1107
softirqs last  enabled at (173766): [<ffffffff814b3f93>] invoke_softirq kernel/softirq.c:445 [inline]
softirqs last  enabled at (173766): [<ffffffff814b3f93>] __irq_exit_rcu+0x123/0x180 kernel/softirq.c:650
softirqs last disabled at (173711): [<ffffffff814b3f93>] invoke_softirq kernel/softirq.c:445 [inline]
softirqs last disabled at (173711): [<ffffffff814b3f93>] __irq_exit_rcu+0x123/0x180 kernel/softirq.c:650
CPU: 0 PID: 3665 Comm: syz-executor189 Not tainted 6.1.0-rc7-syzkaller-00200-gc2bf05db6c78 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
RIP: 0010:csd_lock_wait kernel/smp.c:413 [inline]
RIP: 0010:smp_call_function_many_cond+0x43f/0x10a0 kernel/smp.c:987
Code: e6 e8 95 f7 0a 00 45 85 e4 74 48 48 8b 04 24 49 89 c5 83 e0 07 49 c1 ed 03 49 89 c4 4d 01 fd 41 83 c4 03 e8 c3 fa 0a 00 f3 90 <41> 0f b6 45 00 41 38 c4 7c 08 84 c0 0f 85 4e 0a 00 00 8b 43 08 31
RSP: 0018:ffffc900035177e0 EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff8880b9b3f120 RCX: 0000000000000000
RDX: ffff88807d560000 RSI: ffffffff8174421d RDI: 0000000000000005
RBP: 0000000000000001 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000092080 R12: 0000000000000003
R13: ffffed1017367e25 R14: 0000000000000001 R15: dffffc0000000000
FS:  0000555556ea9300(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1626c6d370 CR3: 00000000793ae000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 on_each_cpu_cond_mask+0x5a/0xa0 kernel/smp.c:1155
 __flush_tlb_multi arch/x86/include/asm/paravirt.h:87 [inline]
 flush_tlb_multi arch/x86/mm/tlb.c:924 [inline]
 flush_tlb_mm_range+0x333/0x4d0 arch/x86/mm/tlb.c:1010
 dup_mmap+0xae1/0x10b0 kernel/fork.c:705
 dup_mm+0x91/0x370 kernel/fork.c:1526
 copy_mm kernel/fork.c:1575 [inline]
 copy_process+0x3bc6/0x7190 kernel/fork.c:2253
 kernel_clone+0xeb/0x980 kernel/fork.c:2671
 __do_sys_clone+0xba/0x100 kernel/fork.c:2812
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f1626bee90b
Code: ed 0f 85 60 01 00 00 64 4c 8b 0c 25 10 00 00 00 45 31 c0 4d 8d 91 d0 02 00 00 31 d2 31 f6 bf 11 00 20 01 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 89 00 00 00 41 89 c5 85 c0 0f 85 90 00 00
RSP: 002b:00007ffe55f753b0 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
RAX: ffffffffffffffda RBX: 00007ffe55f753f0 RCX: 00007f1626bee90b
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000555556ea9300
R10: 0000555556ea95d0 R11: 0000000000000246 R12: 0000000000000000
R13: 00000000000f4240 R14: 0000000000081c66 R15: 00007ffe55f753d4
 </TASK>

Fix bisection attempts:
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets Title
ci-upstream-net-this-kasan-gce 2022/05/24 09:14 net 7fb0269720d7 6b3c5e64 .config console log report syz C
* Struck through repros no longer work on HEAD.
Crashes (811):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets Title
ci-upstream-kasan-gce-selinux-root 2022/12/05 07:42 upstream c2bf05db6c78 e080de16 .config console log report syz C [disk image] [vmlinux] [kernel image] BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/10/14 12:16 upstream 55be6084c8e0 4954e4b2 .config strace log report syz C [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce 2022/08/15 21:07 upstream 7ebfc85e2cd7 8dfcaa3d .config console log report syz C BUG: soft lockup in smp_call_function
ci-upstream-net-this-kasan-gce 2021/12/28 04:31 net 79b69a83705e 6b3c5e64 .config console log report syz C BUG: soft lockup in smp_call_function
ci-upstream-net-this-kasan-gce 2021/10/22 01:10 net e0bfcf9c77d9 c5cb7da8 .config console log report syz C BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2021/10/21 20:33 net-next dfcb63ce1de6 c5cb7da8 .config console log report syz C BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2021/07/17 16:50 net-next ab0441b4a920 f115ae98 .config console log report syz C BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root 2022/10/03 04:13 linux-next aaa11ce2ffc8 feb56351 .config strace log report syz C [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root 2022/08/26 02:07 linux-next b5d939c95186 9b5bf4cd .config strace log report syz C BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/10/31 10:07 upstream b229b6ca5abb 2a71366b .config console log report syz C [disk image] [vmlinux] [kernel image] INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/04/16 15:03 upstream 59250f8a7f3a 8bcc32a6 .config console log report syz C INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2021/06/09 10:47 upstream 4c8684fe555e 5c2fe346 .config console log report syz C INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2020/08/16 01:04 upstream c9c9735c46f5 424dd8e7 .config console log report syz C
ci-upstream-kasan-gce-smack-root 2020/07/22 18:32 upstream 4fa640dc5230 128cd85f .config console log report syz C
ci-upstream-kasan-gce-selinux-root 2020/07/15 07:21 upstream e9919e11e219 609fb517 .config console log report syz C
ci-upstream-kasan-gce-root 2020/07/13 08:57 upstream 4437dd6e8f71 9ebcc5b1 .config console log report syz C
ci-upstream-linux-next-kasan-gce-root 2020/08/12 11:25 linux-next bc09acc9f224 bb3e5fe6 .config console log report syz C
ci-upstream-kasan-gce-smack-root 2022/10/20 21:19 upstream 55be6084c8e0 a0fd4dab .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/10/19 09:41 upstream 55be6084c8e0 b31320fc .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/10/08 19:35 upstream e8bc52cb8df8 aea5da89 .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-root 2022/08/25 21:35 upstream 3f5c20055a64 9b5bf4cd .config console log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-386 2021/11/14 13:28 upstream c8c109546a19 83f5c9b5 .config console log report info BUG: soft lockup in smp_call_function
ci-qemu2-arm32 2021/08/11 20:37 upstream bf152b0b41dc 6972b106 .config console log report info BUG: soft lockup in smp_call_function
ci-upstream-net-this-kasan-gce 2022/01/19 12:43 net 99845220d3c3 0620189b .config console log report info BUG: soft lockup in smp_call_function
ci-upstream-net-kasan-gce 2022/02/15 08:13 net-next 5a8fb33e5305 8b9ca619 .config console log report info BUG: soft lockup in smp_call_function
ci-upstream-linux-next-kasan-gce-root 2022/08/23 06:12 linux-next 05477f3653b8 26a13b38 .config console log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2023/01/18 08:00 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 9598c377d828 42660d9e .config console log report info [disk image] [vmlinux] [kernel image] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2023/01/17 09:30 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 9598c377d828 a63719e7 .config console log report info [disk image] [vmlinux] [kernel image] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/12/20 11:21 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a5541c0811a0 d3e76707 .config console log report info [disk image] [vmlinux] [kernel image] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/12/17 00:06 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a5541c0811a0 05494336 .config console log report info [disk image] [vmlinux] [kernel image] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/12/06 11:16 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci e3cb714fb489 045cbb84 .config console log report info [disk image] [vmlinux] [kernel image] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/12/05 09:23 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci e3cb714fb489 e080de16 .config console log report info [disk image] [vmlinux] [kernel image] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/12/05 04:52 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci e3cb714fb489 e080de16 .config console log report info [disk image] [vmlinux] [kernel image] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/12/04 06:42 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci e3cb714fb489 e080de16 .config console log report info [disk image] [vmlinux] [kernel image] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/11/02 06:46 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci bbed346d5a96 edac4fd1 .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/10/25 05:56 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci bbed346d5a96 45645420 .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/10/23 23:26 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci bbed346d5a96 23bf86af .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/10/21 12:13 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci bbed346d5a96 4bfd3c27 .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/10/03 19:41 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci bbed346d5a96 feb56351 .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/27 23:41 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci c194837ebb57 75c78242 .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/23 23:13 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci c194837ebb57 0042f2b4 .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/21 04:37 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 5aa266bb455b c4b8ccfd .config console log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/20 08:47 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a6b443748715 7c41a9ba .config console log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/19 11:14 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a6b443748715 dd9a85ff .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/19 07:22 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a6b443748715 dd9a85ff .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/10 22:15 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a6b443748715 356d8217 .config console log report info [disk image] [vmlinux] BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/05 22:57 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 85413d1e802e 9dcd38fc .config console log report info BUG: soft lockup in smp_call_function
ci-upstream-gce-arm64 2022/09/04 22:20 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 85413d1e802e 28811d0a .config console log report info BUG: soft lockup in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/10/27 02:27 upstream 98555239e4c3 86777b7f .config console log report info [disk image] [vmlinux] [kernel image] INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce 2022/10/26 09:09 upstream 4dc12f37a8e9 1984aebd .config console log report info [disk image] [vmlinux] [kernel image] INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/10/06 01:16 upstream 833477fce7a1 2c6543ad .config console log report info [disk image] [vmlinux] INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/09/25 15:57 upstream 105a36f3694e 0042f2b4 .config console log report info [disk image] [vmlinux] INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/09/13 10:30 upstream e839a756012b a08652b0 .config console log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/09/12 13:54 upstream 80e78fcce86d 356d8217 .config console log report info [disk image] [vmlinux] INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/09/12 11:10 upstream 80e78fcce86d 356d8217 .config console log report info [disk image] [vmlinux] INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/09/02 05:18 upstream 42e66b1cc3a0 a805568e .config console log report info [disk image] [vmlinux] INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/08/30 16:01 upstream dcf8e5633e2e 4a380809 .config console log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/08/22 18:39 upstream 1c23f9e627a7 26a13b38 .config console log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/07/23 01:06 upstream 4ba1329cbb94 22343af4 .config console log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/07/22 04:48 upstream 68e77ffbfd06 5e6028b9 .config console log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-root 2022/07/15 11:32 upstream e5d523f1ae8f 5d921b08 .config console log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-selinux-root 2022/07/15 03:35 upstream 9bd572ec7a66 5d921b08 .config console log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2022/06/14 16:15 upstream b13baccc3850 127d1faf .config console log report info INFO: rcu detected stall in smp_call_function
ci-upstream-kasan-gce-smack-root 2021/01/17 04:44 upstream 0da0a8a0a0e1 65a7a854 .config console log report info
ci-upstream-kasan-gce-root 2020/07/12 23:01 upstream 4437dd6e8f71 9ebcc5b1 .config console log report
ci-upstream-kasan-gce-smack-root 2020/07/10 22:54 upstream a581387e415b 18d18b59 .config console log report
ci-upstream-kasan-gce-selinux-root 2020/07/08 22:22 upstream 0bddd227f3dc bc238812 .config console log report
ci-upstream-bpf-kasan-gce 2021/09/03 00:40 bpf 57f780f1c433 15cea0a3 .config console log report info INFO: rcu detected stall in smp_call_function
ci-upstream-bpf-next-kasan-gce 2020/11/02 15:12 bpf-next cb5dc5b062a9 8bc4594f .config console log report info
ci-upstream-linux-next-kasan-gce-root 2022/09/27 13:58 linux-next aaa11ce2ffc8 87840e00 .config console log report info [disk image] [vmlinux] INFO: rcu detected stall in smp_call_function
ci-upstream-linux-next-kasan-gce-root 2022/08/23 19:53 linux-next 05477f3653b8 cea8b0f7 .config console log report info INFO: rcu detected stall in smp_call_function
ci-upstream-linux-next-kasan-gce-root 2022/06/15 15:28 linux-next 6012273897fe 1719ee24 .config console log report info INFO: rcu detected stall in smp_call_function
* Struck through repros no longer work on HEAD.