syzbot


BUG: soft lockup in smp_call_function

Status: upstream: reported C repro on 2023/03/19 12:10
Bug presence: origin:upstream
Labels: missing-backport
[Documentation on labels]
Reported-by: syzbot+f61e8f4a149eefab5677@syzkaller.appspotmail.com
First crash: 451d, last: 16h47m
Bug presence (3)
Date Name Commit Repro Result
2023/08/06 linux-5.15.y (ToT) 38d4ca22a528 C [report] no output from test machine
2023/06/22 upstream (ToT) dad9774deaf1 C [report] BUG: workqueue lockup
2023/08/06 upstream (ToT) f0ab9f34e59e C Didn't crash
Similar bugs (10)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
android-6-1 BUG: soft lockup in smp_call_function origin:upstream C 52 1d15h 71d 0/2 upstream: reported C repro on 2024/04/02 06:05
linux-4.19 BUG: soft lockup in smp_call_function C error 2 653d 781d 0/1 upstream: reported C repro on 2022/04/23 05:26
upstream BUG: soft lockup in smp_call_function kernel C done 1123 10h43m 1431d 0/28 upstream: reported C repro on 2020/07/12 23:02
linux-6.1 BUG: soft lockup in smp_call_function 58 6d17h 453d 0/3 upstream: reported on 2023/03/18 00:36
android-54 BUG: soft lockup in smp_call_function 1 950d 950d 0/2 auto-closed as invalid on 2022/03/05 18:04
linux-4.14 BUG: soft lockup in smp_call_function C 1 479d 781d 0/1 upstream: reported C repro on 2022/04/23 04:43
android-5-15 BUG: soft lockup in smp_call_function origin:upstream C error inconclusive 528 44m 465d 0/2 upstream: reported C repro on 2023/03/05 21:20
android-54 BUG: soft lockup in smp_call_function (2) C 23 12d 136d 0/2 upstream: reported C repro on 2024/01/28 03:58
android-5-10 BUG: soft lockup in smp_call_function C error 67 10d 326d 0/2 upstream: reported C repro on 2023/07/22 22:59
upstream INFO: rcu detected stall in smp_call_function perf C 107 1159d 1414d 20/28 fixed on 2021/04/13 06:26
Fix bisection attempts (5)
Created Duration User Patch Repo Result
2024/05/16 21:05 1m fix candidate upstream error job log (0)
2024/03/26 04:25 3m fix candidate upstream error job log (0)
2024/01/21 18:14 0m fix candidate upstream error job log (0)
2023/12/17 05:33 0m fix candidate upstream error job log (0)
2023/11/12 14:10 1m fix candidate upstream error job log (0)

Sample crash report:
watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [syz-executor247:4047]
Modules linked in:
irq event stamp: 9390
hardirqs last  enabled at (9389): [<ffff800011952c58>] __exit_to_kernel_mode arch/arm64/kernel/entry-common.c:81 [inline]
hardirqs last  enabled at (9389): [<ffff800011952c58>] exit_to_kernel_mode+0x100/0x178 arch/arm64/kernel/entry-common.c:91
hardirqs last disabled at (9390): [<ffff800011952ea0>] enter_el1_irq_or_nmi+0x10/0x1c arch/arm64/kernel/entry-common.c:227
softirqs last  enabled at (4938): [<ffff800008020ccc>] softirq_handle_end kernel/softirq.c:401 [inline]
softirqs last  enabled at (4938): [<ffff800008020ccc>] __do_softirq+0xb5c/0xe20 kernel/softirq.c:587
softirqs last disabled at (4753): [<ffff8000081b573c>] do_softirq_own_stack include/asm-generic/softirq_stack.h:10 [inline]
softirqs last disabled at (4753): [<ffff8000081b573c>] invoke_softirq kernel/softirq.c:439 [inline]
softirqs last disabled at (4753): [<ffff8000081b573c>] __irq_exit_rcu+0x28c/0x534 kernel/softirq.c:636
CPU: 1 PID: 4047 Comm: syz-executor247 Not tainted 5.15.118-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/25/2023
pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : csd_lock_wait kernel/smp.c:440 [inline]
pc : smp_call_function_many_cond+0xa50/0x1088 kernel/smp.c:969
lr : csd_lock_wait kernel/smp.c:440 [inline]
lr : smp_call_function_many_cond+0xa68/0x1088 kernel/smp.c:969
sp : ffff80001d047900
x29: ffff80001d047950 x28: ffff0001b482cd48 x27: dfff800000000000
x26: 0000000000000011 x25: ffff0001b482cd40 x24: ffff8000148f0ff8
x23: 1fffe000369059a8 x22: ffff0001b48131c8 x21: ffff800014adda40
x20: 0000000000000000 x19: ffff0001b482cd48 x18: ffff80001d047620
x17: 0000000000000000 x16: ffff8000119547c4 x15: 0000000000011af3
x14: 1ffff0000291e06a x13: ffffffffffffffff x12: 0000000000000000
x11: ff8080000840ef98 x10: 0000000000000000 x9 : 0000000000000000
x8 : 0000000000000011 x7 : 0000000000000000 x6 : 0000000000000000
x5 : 0000000000000080 x4 : 0000000000000000 x3 : ffff80000840ef8c
x2 : 0000000000000000 x1 : 0000000000000004 x0 : 0000000000000001
Call trace:
 __cmpwait_case_32 arch/arm64/include/asm/cmpxchg.h:252 [inline]
 __cmpwait arch/arm64/include/asm/cmpxchg.h:278 [inline]
 csd_lock_wait kernel/smp.c:440 [inline]
 smp_call_function_many_cond+0xa50/0x1088 kernel/smp.c:969
 smp_call_function_many+0x40/0x54 kernel/smp.c:993
 clock_was_set+0x69c/0x784 kernel/time/hrtimer.c:974
 do_settimeofday64+0x6e0/0x8f0 kernel/time/timekeeping.c:1328
 do_sys_settimeofday64+0x1b8/0x23c kernel/time/time.c:195
 posix_clock_realtime_set+0x24/0x34 kernel/time/posix-timers.c:185
 __do_sys_clock_settime kernel/time/posix-timers.c:1092 [inline]
 __se_sys_clock_settime kernel/time/posix-timers.c:1080 [inline]
 __arm64_sys_clock_settime+0x1fc/0x248 kernel/time/posix-timers.c:1080
 __invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
 invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:52
 el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
 do_el0_svc+0x58/0x14c arch/arm64/kernel/syscall.c:181
 el0_svc+0x7c/0x1f0 arch/arm64/kernel/entry-common.c:596
 el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:614
 el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584

Crashes (97):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/06/22 05:03 linux-5.15.y f67653019430 79782afc .config console log report syz C [disk image] [vmlinux] [kernel image] [mounted in repro] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/01/14 06:19 linux-5.15.y 26c690eff0a5 551587c1 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf INFO: rcu detected stall in smp_call_function
2024/01/13 23:54 linux-5.15.y 26c690eff0a5 551587c1 .config console log report syz [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf INFO: rcu detected stall in smp_call_function
2024/04/01 00:39 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan BUG: soft lockup in smp_call_function
2023/11/20 14:56 linux-5.15.y 2a910f4af54d cb976f63 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan BUG: soft lockup in smp_call_function
2024/05/28 23:41 linux-5.15.y c61bd26ae81a 34889ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/05/27 04:31 linux-5.15.y c61bd26ae81a a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/05/26 16:05 linux-5.15.y c61bd26ae81a a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/05/21 11:54 linux-5.15.y 83655231580b 4c0d3ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/05/19 16:08 linux-5.15.y 83655231580b c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/05/17 10:02 linux-5.15.y 284087d4f7d5 c2e07261 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/05/14 05:42 linux-5.15.y 284087d4f7d5 fdb4c10c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/05/12 01:34 linux-5.15.y 284087d4f7d5 9026e142 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/05/10 09:32 linux-5.15.y 284087d4f7d5 de979bc2 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/05/02 08:40 linux-5.15.y b925f60c6ee7 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/04/14 16:48 linux-5.15.y fa3df276cd36 c8349e48 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/04/12 21:32 linux-5.15.y cdfd0a7f0139 c8349e48 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/04/12 17:21 linux-5.15.y cdfd0a7f0139 c8349e48 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/04/10 13:39 linux-5.15.y 9465fef4ae35 4320ec32 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/04/10 07:12 linux-5.15.y 9465fef4ae35 171ec371 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/04/09 00:34 linux-5.15.y 9465fef4ae35 53df08b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/04/08 19:28 linux-5.15.y 9465fef4ae35 53df08b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/04/06 14:14 linux-5.15.y 9465fef4ae35 ca620dd8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/04/03 07:17 linux-5.15.y 9465fef4ae35 7925100d .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/04/02 07:30 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/03/28 13:49 linux-5.15.y 9465fef4ae35 e91187ee .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/03/26 19:22 linux-5.15.y b95c01af2113 454571b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/03/05 17:41 linux-5.15.y 80efc6265290 f39a7eed .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/02/29 05:15 linux-5.15.y 458ce51d0356 352ab904 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/02/27 06:30 linux-5.15.y 458ce51d0356 05e69c83 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/02/19 19:23 linux-5.15.y 6139f2a02fe0 3af7dd65 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/01/24 17:24 linux-5.15.y ddcaf4999061 1e153dc8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/01/12 20:55 linux-5.15.y 26c690eff0a5 551587c1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/01/09 01:06 linux-5.15.y 26c690eff0a5 4c0fd4bb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/01/07 17:17 linux-5.15.y 26c690eff0a5 d0304e9c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/12/06 20:47 linux-5.15.y 9b91d36ba301 e3299f55 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/11/10 14:59 linux-5.15.y 80529b4968a8 45e9b83e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/11/09 15:08 linux-5.15.y 80529b4968a8 56230772 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/11/03 22:42 linux-5.15.y 12952a23a5da 500bfdc4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/10/10 22:58 linux-5.15.y 02e21884dcf2 83165b57 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/10/10 07:09 linux-5.15.y 1edcec18cfb7 c9be5398 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/10/09 20:33 linux-5.15.y 1edcec18cfb7 3c53c7d9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/10/06 19:38 linux-5.15.y 1edcec18cfb7 ea12a918 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/05/09 04:07 linux-5.15.y 8a7f2a5c5aa1 c7a5e2a0 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan BUG: soft lockup in smp_call_function
2023/03/19 12:10 linux-5.15.y 8020ae3c051d 7939252e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan BUG: soft lockup in smp_call_function
2023/08/24 05:47 linux-5.15.y f6f7927ac664 4d7ae7ab .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/08/22 18:12 linux-5.15.y f6f7927ac664 b81ca3f6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/04/19 12:01 linux-5.15.y 4fdad925aa1a 94b4184e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2023/03/23 00:48 linux-5.15.y 115472395b0a f94b4a29 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in smp_call_function
2024/06/12 09:13 linux-5.15.y c61bd26ae81a 4d75f4f7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in smp_call_function
2024/06/10 19:41 linux-5.15.y c61bd26ae81a 048c640a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in smp_call_function
2024/04/20 17:57 linux-5.15.y c52b9710c83d af24b050 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in smp_call_function
2024/03/20 09:43 linux-5.15.y b95c01af2113 a485f239 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in smp_call_function
2024/01/08 06:04 linux-5.15.y 26c690eff0a5 d0304e9c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf INFO: rcu detected stall in smp_call_function
2023/12/06 17:47 linux-5.15.y 9b91d36ba301 e3299f55 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in smp_call_function
2023/10/04 19:04 linux-5.15.y b911329317b4 b7d7ff54 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in smp_call_function
2023/09/07 04:02 linux-5.15.y aff03380bda4 72324844 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in smp_call_function
* Struck through repros no longer work on HEAD.