syzbot

 sign-in | mailing list | source | docs
🐞 Open [1583]
≡ Subsystems
🐞 Fixed [6167]
🐞 Invalid [15458]
⬇ Missing Backports [174]
≡ Crashes
Kernel Health Bugs/Month Bug Lifetimes Fuzzing
Total Repo Heatmap Subsystems Heatmap
💬 Send us feedback

BUG: soft lockup in __hrtimer_run_queues (4)

Status: moderation: reported on 2025/04/06 00:05
Subsystems: kernel
[Documentation on labels]
Reported-by: syzbot+623c3b3ba193ed23e24f@syzkaller.appspotmail.com
First crash: 6d21h, last: 6d21h
▶ ▼ Similar bugs (4)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream BUG: soft lockup in __hrtimer_run_queues (2) kernel 1 282d 278d 0/28 auto-obsoleted due to no activity on 2024/09/27 22:42
upstream BUG: soft lockup in __hrtimer_run_queues (3) kernel 1 99d 95d 0/28 auto-obsoleted due to no activity on 2025/03/30 03:09
upstream BUG: soft lockup in __hrtimer_run_queues kernel 2 1544d 1589d 0/28 auto-closed as invalid on 2021/04/15 12:33
upstream INFO: rcu detected stall in __hrtimer_run_queues kernel C inconclusive done 22 961d 1508d 0/28 auto-obsoleted due to no activity on 2022/12/19 03:45

Sample crash report:
watchdog: BUG: soft lockup - CPU#0 stuck for 115s! [syz.4.212:6534]
Modules linked in:
irq event stamp: 23074863
hardirqs last  enabled at (23074862): [<ffffffff8c2dcf13>] irqentry_exit+0x63/0x90 kernel/entry/common.c:357
hardirqs last disabled at (23074863): [<ffffffff8c2dab0e>] sysvec_apic_timer_interrupt+0xe/0xc0 arch/x86/kernel/apic/apic.c:1049
softirqs last  enabled at (22845748): [<ffffffff8184596b>] __do_softirq kernel/softirq.c:613 [inline]
softirqs last  enabled at (22845748): [<ffffffff8184596b>] invoke_softirq kernel/softirq.c:453 [inline]
softirqs last  enabled at (22845748): [<ffffffff8184596b>] __irq_exit_rcu+0xfb/0x220 kernel/softirq.c:680
softirqs last disabled at (22845751): [<ffffffff8184596b>] __do_softirq kernel/softirq.c:613 [inline]
softirqs last disabled at (22845751): [<ffffffff8184596b>] invoke_softirq kernel/softirq.c:453 [inline]
softirqs last disabled at (22845751): [<ffffffff8184596b>] __irq_exit_rcu+0xfb/0x220 kernel/softirq.c:680
CPU: 0 UID: 0 PID: 6534 Comm: syz.4.212 Not tainted 6.14.0-syzkaller-g1e7857b28020 #0 PREEMPT(full) 
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2025
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0xd9/0x140 kernel/locking/spinlock.c:194
Code: 9c 8f 44 24 20 42 80 3c 2b 00 74 08 4c 89 f7 e8 8d 1d fc f5 f6 44 24 21 02 75 55 41 f7 c7 00 02 00 00 74 01 fb bf 01 00 00 00 <e8> c2 18 60 f5 65 8b 05 8b 96 36 07 85 c0 74 46 48 c7 04 24 0e 36
RSP: 0018:ffffc90000007bc0 EFLAGS: 00000206
RAX: 6a047783e7b2fb00 RBX: 1ffff92000000f7c RCX: ffffffff81cb6ebc
RDX: 0000000000000000 RSI: ffffffff8e691f31 RDI: 0000000000000001
RBP: ffffc90000007c50 R08: ffffffff905f5377 R09: 1ffffffff20bea6e
R10: dffffc0000000000 R11: fffffbfff20bea6f R12: 1ffff92000000f78
R13: dffffc0000000000 R14: ffffc90000007be0 R15: 0000000000000246
FS:  0000000000000000(0000) GS:ffff888124faf000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f61339a6038 CR3: 0000000012906000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
Call Trace:
 <IRQ>
 __run_hrtimer kernel/time/hrtimer.c:1787 [inline]
 __hrtimer_run_queues+0x482/0xd40 kernel/time/hrtimer.c:1855
 hrtimer_run_softirq+0x19a/0x2c0 kernel/time/hrtimer.c:1872
 handle_softirqs+0x2d6/0x9b0 kernel/softirq.c:579
 __do_softirq kernel/softirq.c:613 [inline]
 invoke_softirq kernel/softirq.c:453 [inline]
 __irq_exit_rcu+0xfb/0x220 kernel/softirq.c:680
 irq_exit_rcu+0x9/0x30 kernel/softirq.c:696
 instr_sysvec_irq_work arch/x86/kernel/irq_work.c:17 [inline]
 sysvec_irq_work+0xa3/0xc0 arch/x86/kernel/irq_work.c:17
 </IRQ>
 <TASK>
 asm_sysvec_irq_work+0x1a/0x20 arch/x86/include/asm/idtentry.h:738
RIP: 0010:preempt_schedule_irq+0xf9/0x1c0 kernel/sched/core.c:7090
Code: 89 fe 49 c1 ee 03 eb 0d 48 f7 03 08 00 00 00 0f 84 89 00 00 00 bf 01 00 00 00 e8 72 a5 61 f5 e8 6d 84 9c f5 fb bf 01 00 00 00 <e8> 32 a5 ff ff 43 80 3c 26 00 74 08 4c 89 ff e8 93 ac fd f5 48 c7
RSP: 0018:ffffc9001aae70c0 EFLAGS: 00000286
RAX: 6a047783e7b2fb00 RBX: 1ffff9200355ce20 RCX: ffffffff81cb6ebc
RDX: 0000000000000000 RSI: ffffffff8e691f31 RDI: 0000000000000001
RBP: ffffc9001aae7180 R08: ffffffff905f5377 R09: 1ffffffff20bea6e
R10: dffffc0000000000 R11: fffffbfff20bea6f R12: dffffc0000000000
R13: 1ffff9200355ce18 R14: 1ffff9200355ce1c R15: ffffc9001aae70e0
 irqentry_exit+0x5e/0x90 kernel/entry/common.c:354
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:rcu_preempt_read_enter kernel/rcu/tree_plugin.h:391 [inline]
RIP: 0010:__rcu_read_lock+0x30/0xb0 kernel/rcu/tree_plugin.h:414
Code: 57 41 56 53 49 be 00 00 00 00 00 fc ff df 65 4c 8b 3c 25 08 10 67 93 49 81 c7 44 04 00 00 4c 89 fb 48 c1 eb 03 42 0f b6 04 33 <84> c0 75 35 41 8b 2f ff c5 42 0f b6 04 33 84 c0 75 3e 41 89 2f 42
RSP: 0018:ffffc9001aae7240 EFLAGS: 00000a03
RAX: 0000000000000000 RBX: 1ffff11006001bc8 RCX: 0000000000000001
RDX: dffffc0000000000 RSI: ffffc9001aae8000 RDI: ffffc9001aae73a0
RBP: ffffc9001aae73d8 R08: ffffc9001aae7301 R09: 0000000000000000
R10: ffffc9001aae73a0 R11: fffff5200355ce80 R12: ffffc9001aae73a0
R13: dffffc0000000000 R14: dffffc0000000000 R15: ffff88803000de44
 rcu_read_lock include/linux/rcupdate.h:839 [inline]
 class_rcu_constructor include/linux/rcupdate.h:1155 [inline]
 unwind_next_frame+0xb1/0x23b0 arch/x86/kernel/unwind_orc.c:479
 __unwind_start+0x59a/0x740 arch/x86/kernel/unwind_orc.c:758
 unwind_start arch/x86/include/asm/unwind.h:64 [inline]
 arch_stack_walk+0xe7/0x150 arch/x86/kernel/stacktrace.c:24
 stack_trace_save+0x11a/0x1d0 kernel/stacktrace.c:122
 kasan_save_stack+0x3f/0x60 mm/kasan/common.c:47
 kasan_record_aux_stack+0xbf/0xd0 mm/kasan/generic.c:548
 task_work_add+0xc1/0x490 kernel/task_work.c:65
 mntput_no_expire+0x5ae/0x880 fs/namespace.c:1514
 mntput fs/namespace.c:1531 [inline]
 namespace_unlock+0x704/0x7a0 fs/namespace.c:1824
 drop_collected_mounts fs/namespace.c:2425 [inline]
 put_mnt_ns+0xdc/0x120 fs/namespace.c:6231
 free_nsproxy+0x4d/0x3c0 kernel/nsproxy.c:193
 do_exit+0xa0c/0x2950 kernel/exit.c:953
 do_group_exit+0x207/0x2c0 kernel/exit.c:1103
 get_signal+0x1696/0x1730 kernel/signal.c:3035
 arch_do_signal_or_restart+0x98/0x840 arch/x86/kernel/signal.c:337
 exit_to_user_mode_loop kernel/entry/common.c:111 [inline]
 exit_to_user_mode_prepare include/linux/entry-common.h:329 [inline]
 __syscall_exit_to_user_mode_work kernel/entry/common.c:207 [inline]
 syscall_exit_to_user_mode+0xce/0x340 kernel/entry/common.c:218
 do_syscall_64+0x100/0x230 arch/x86/entry/syscall_64.c:100
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f909cb8d169
Code: Unable to access opcode bytes at 0x7f909cb8d13f.
RSP: 002b:00007f909d970038 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: fffffffffffffffc RBX: 00007f909cda6080 RCX: 00007f909cb8d169
RDX: 0000000000000050 RSI: 0000200000000600 RDI: 000000000000000a
RBP: 00007f909cc0e2a0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f909cda6080 R15: 00007ffe3237e048
 </TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 53 Comm: kworker/u8:3 Not tainted 6.14.0-syzkaller-g1e7857b28020 #0 PREEMPT(full) 
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2025
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:rep_nop arch/x86/include/asm/vdso/processor.h:13 [inline]
RIP: 0010:cpu_relax arch/x86/include/asm/vdso/processor.h:18 [inline]
RIP: 0010:csd_lock_wait kernel/smp.c:340 [inline]
RIP: 0010:smp_call_function_many_cond+0x1baa/0x2d40 kernel/smp.c:885
Code: b6 04 03 84 c0 75 7e 45 8b 65 00 44 89 e6 83 e6 01 31 ff e8 58 02 0c 00 41 83 e4 01 4c 8b 64 24 68 75 07 e8 08 fe 0b 00 eb 41 <f3> 90 48 b8 00 00 00 00 00 fc ff df 0f b6 04 03 84 c0 75 11 41 f7
RSP: 0018:ffffc90000bf7640 EFLAGS: 00000293
RAX: ffffffff81b780bd RBX: 1ffff110170c823d RCX: ffff888021ac0000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90000bf7840 R08: ffffffff81b78088 R09: 1ffffffff20bea6e
R10: dffffc0000000000 R11: fffffbfff20bea6f R12: ffff8880b873ad08
R13: ffff8880b86411e8 R14: ffff8880b873ad00 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8881250af000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7fdf305020 CR3: 000000000eb38000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
Call Trace:
 <TASK>
 on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1052
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:2455 [inline]
 text_poke_bp_batch+0x354/0xb30 arch/x86/kernel/alternative.c:2665
 text_poke_flush arch/x86/kernel/alternative.c:2856 [inline]
 text_poke_finish+0x30/0x50 arch/x86/kernel/alternative.c:2863
 arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
 static_key_enable_cpuslocked+0x136/0x260 kernel/jump_label.c:210
 static_key_enable+0x1a/0x20 kernel/jump_label.c:223
 toggle_allocation_gate+0xc0/0x250 mm/kfence/core.c:850
 process_one_work kernel/workqueue.c:3238 [inline]
 process_scheduled_works+0xac3/0x18e0 kernel/workqueue.c:3319
 worker_thread+0x870/0xd50 kernel/workqueue.c:3400
 kthread+0x7b7/0x940 kernel/kthread.c:464
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:153
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
 </TASK>

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2025/04/01 23:57 bpf-next 1e7857b28020 d863cf4b .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-bpf-next-kasan-gce BUG: soft lockup in __hrtimer_run_queues
* Struck through repros no longer work on HEAD.