INFO: rcu detected stall in __hrtimer_run_queues

Status: auto-obsoleted due to no activity on 2022/12/19 03:45
Subsystems: kernel
[Documentation on labels]
Reported-by: syzbot+de9526ade17c659d8336@syzkaller.appspotmail.com
First crash: 2127d, last: 1452d
Cause bisection: the issue happens on the oldest tested release (bisect log)
Crash: INFO: rcu detected stall in mac80211_hwsim_beacon (log)
Repro: C syz .config
  
Fix bisection: fixed by (bisect log) :
commit b60876296847e6cd7f1da4b8b7f0f31399d59aa1
Author: Jens Axboe <axboe@kernel.dk>
Date: Fri Oct 15 21:03:52 2021 +0000

  block: improve layout of struct request

  
▶ ▼ Discussions (1)
Title Replies (including bot) Last reply
INFO: rcu detected stall in __hrtimer_run_queues 2 (4) 2021/11/17 11:59
▶ ▼ Similar bugs (6)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 INFO: rcu detected stall in __hrtimer_run_queues 1 1 4d23h 4d23h 0/3 upstream: reported on 2026/08/07 07:45
upstream INFO: rcu detected stall in __hrtimer_run_queues (3) prio:high kernel 1 syz 20 3d19h 3d11h 0/29 upstream: reported syz repro on 2026/08/08 20:09
upstream INFO: rcu detected stall in __hrtimer_run_queues (2) kernel 1 1 406d 406d 0/29 auto-obsoleted due to no activity on 2025/09/29 21:02
upstream BUG: soft lockup in __hrtimer_run_queues (4) kernel 1 1 497d 493d 0/29 auto-obsoleted due to no activity on 2025/06/30 23:58
upstream BUG: soft lockup in __hrtimer_run_queues (2) kernel 1 1 773d 769d 0/29 auto-obsoleted due to no activity on 2024/09/27 22:42
upstream BUG: soft lockup in __hrtimer_run_queues (3) kernel 1 1 590d 586d 0/29 auto-obsoleted due to no activity on 2025/03/30 03:09
▶ ▼ Last patch testing requests (7)
Created Duration User Patch Repo Result
2022/11/29 04:30 21m retest repro upstream no crash log
2022/07/31 11:01 10m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git f40ddce8 report log
2022/07/31 09:36 15m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git f40ddce8 report log
2022/07/31 08:27 15m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git f40ddce8 report log
2022/07/31 08:03 15m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git f40ddce8 report log
2022/07/31 07:25 10m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git f40ddce8 report log
2022/03/27 09:26 16m hdanton@sina.com upstream no crash
▶ ▼ Fix bisection attempts (7)
Created Duration User Patch Repo Result
2021/11/16 10:16 5h24m bisect fix upstream OK (1) job log
2021/10/17 09:15 36m bisect fix upstream OK (0) job log log
2021/08/23 03:15 36m bisect fix upstream OK (0) job log log
2021/07/24 02:38 36m bisect fix upstream OK (0) job log log
2021/06/23 20:45 38m bisect fix upstream OK (0) job log log
2021/05/24 20:12 32m bisect fix upstream OK (0) job log log
2021/03/25 19:52 28m bisect fix upstream OK (0) job log log
▶ ▼ Cause bisection attempts (2)
Created Duration User Patch Repo Result
2021/02/27 11:06 3h52m bisect upstream OK (0) job log log
2021/02/16 20:47 0m bisect upstream error job log

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-....: (10153 ticks this GP) idle=f1a/1/0x4000000000000000 softirq=10867/10868 fqs=925 
	(t=10502 jiffies g=10029 q=19103)
NMI backtrace for cpu 1
CPU: 1 PID: 10530 Comm: syz-executor248 Not tainted 5.11.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x107/0x163 lib/dump_stack.c:120
 nmi_cpu_backtrace.cold+0x44/0xd7 lib/nmi_backtrace.c:105
 nmi_trigger_cpumask_backtrace+0x1b3/0x230 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x1f4/0x230 kernel/rcu/tree_stall.h:337
 print_cpu_stall kernel/rcu/tree_stall.h:569 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:643 [inline]
 rcu_pending kernel/rcu/tree.c:3751 [inline]
 rcu_sched_clock_irq.cold+0x48e/0xedf kernel/rcu/tree.c:2580
 update_process_times+0x16d/0x200 kernel/time/timer.c:1782
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1369
 __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
 __hrtimer_run_queues+0x1c0/0xe40 kernel/time/hrtimer.c:1583
 hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
 run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:91 [inline]
 sysvec_apic_timer_interrupt+0x48/0x100 arch/x86/kernel/apic/apic.c:1100
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:629
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:161 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x50 kernel/locking/spinlock.c:191
Code: f8 5d c3 66 90 55 48 89 fd 48 83 c7 18 53 48 89 f3 48 8b 74 24 10 e8 9a d6 5b f8 48 89 ef e8 42 8b 5c f8 f6 c7 02 75 1a 53 9d <bf> 01 00 00 00 e8 81 92 50 f8 65 8b 05 7a f8 04 77 85 c0 74 0a 5b
RSP: 0018:ffffc90000db0e48 EFLAGS: 00000286
RAX: 0000000000ce281a RBX: 0000000000000286 RCX: 1ffffffff1b46a19
RDX: 0000000000000000 RSI: 0000000000000102 RDI: 0000000000000000
RBP: ffff8880b9d26a00 R08: 0000000000000001 R09: 0000000000000001
R10: ffffffff8178a8b8 R11: 0000000000000000 R12: 000000a80fcc296a
R13: ffff8880b9d26c80 R14: ffff8880b9d26a00 R15: ffffffff851589e0
 __run_hrtimer kernel/time/hrtimer.c:1515 [inline]
 __hrtimer_run_queues+0x51a/0xe40 kernel/time/hrtimer.c:1583
 hrtimer_run_softirq+0x17b/0x360 kernel/time/hrtimer.c:1600
 __do_softirq+0x29b/0x9f6 kernel/softirq.c:343
 asm_call_irq_on_stack+0xf/0x20
 </IRQ>
 __run_on_irqstack arch/x86/include/asm/irq_stack.h:26 [inline]
 run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:77 [inline]
 do_softirq_own_stack+0xaa/0xd0 arch/x86/kernel/irq_64.c:77
 invoke_softirq kernel/softirq.c:226 [inline]
 __irq_exit_rcu kernel/softirq.c:420 [inline]
 irq_exit_rcu+0x134/0x200 kernel/softirq.c:432
 sysvec_apic_timer_interrupt+0x4d/0x100 arch/x86/kernel/apic/apic.c:1100
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:629
RIP: 0010:queue_work_on+0x83/0xd0 kernel/workqueue.c:1530
Code: 31 ff 89 ee e8 6e 02 29 00 40 84 ed 74 46 e8 e4 fb 28 00 31 ff 48 89 de e8 ca 03 29 00 48 85 db 75 26 e8 d0 fb 28 00 41 56 9d <48> 83 c4 08 44 89 f8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 e8 b6 fb 28
RSP: 0018:ffffc90002e5fc80 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 0000000000000200 RCX: 0000000000000000
RDX: ffff8880265f5340 RSI: ffffffff8149da20 RDI: 0000000000000000
RBP: ffffc90002e68000 R08: 0000000000000001 R09: 0000000000000001
R10: ffffffff8178a8b8 R11: 0000000000000000 R12: ffff8880b9d31568
R13: ffff888010c64c00 R14: 0000000000000293 R15: 0000000000000001
 queue_work include/linux/workqueue.h:507 [inline]
 schedule_work include/linux/workqueue.h:568 [inline]
 __vfree_deferred mm/vmalloc.c:2307 [inline]
 vfree_atomic+0xac/0xe0 mm/vmalloc.c:2325
 free_thread_stack kernel/fork.c:291 [inline]
 release_task_stack kernel/fork.c:428 [inline]
 put_task_stack+0x29c/0x480 kernel/fork.c:439
 finish_task_switch.isra.0+0x557/0x7e0 kernel/sched/core.c:4236
 context_switch kernel/sched/core.c:4330 [inline]
 __schedule+0x914/0x21a0 kernel/sched/core.c:5078
 preempt_schedule_common+0x45/0xc0 kernel/sched/core.c:5238
 preempt_schedule_thunk+0x16/0x18 arch/x86/entry/thunk_64.S:35
 __raw_spin_unlock include/linux/spinlock_api_smp.h:152 [inline]
 _raw_spin_unlock+0x36/0x40 kernel/locking/spinlock.c:183
 spin_unlock include/linux/spinlock.h:394 [inline]
 pick_file+0x129/0x1e0 fs/file.c:613
 close_fd+0x44/0x80 fs/file.c:622
 __do_sys_close fs/open.c:1299 [inline]
 __se_sys_close fs/open.c:1297 [inline]
 __x64_sys_close+0x2f/0xa0 fs/open.c:1297
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x403353
Code: c7 c2 c0 ff ff ff f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8
RSP: 002b:00007ffdb38739a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 0000000000403353
RDX: 0000000000042000 RSI: 0000000000000004 RDI: 0000000000000003
RBP: 0000000000000000 R08: 0000000000000031 R09: 0000000000000031
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000000aa47b
R13: 00007ffdb3873a00 R14: 00007ffdb38739f0 R15: 00007ffdb38739c4

Crashes (22):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/02/16 20:47 upstream f40ddce88593 98682e5e .config console log report syz C ci-upstream-kasan-gce-root INFO: rcu detected stall in __hrtimer_run_queues
2021/11/22 03:29 upstream 40c93d7fff6f 4eb20a4e .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in __hrtimer_run_queues
2021/09/16 22:23 upstream ff1ffd71d5f0 aae492f2 .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in __hrtimer_run_queues
2021/04/24 20:12 upstream 8db5efb83fa9 17f0b706 .config console log report info ci-upstream-kasan-gce-root INFO: rcu detected stall in __hrtimer_run_queues
2021/02/23 19:51 upstream 3b9cdafb5358 fcc6d71b .config console log report info ci-upstream-kasan-gce-root INFO: rcu detected stall in __hrtimer_run_queues
2021/02/22 09:19 upstream 31caf8b2a847 a659b3f1 .config console log report info ci-upstream-kasan-gce-root INFO: rcu detected stall in __hrtimer_run_queues
2021/02/22 08:41 upstream 31caf8b2a847 a659b3f1 .config console log report info ci-upstream-kasan-gce-root INFO: rcu detected stall in __hrtimer_run_queues
2021/01/29 12:14 upstream bec4c2968fce 6593fd32 .config console log report info ci-upstream-kasan-gce-root INFO: rcu detected stall in __hrtimer_run_queues
2022/08/21 03:44 linux-next 8755ae45a9e8 26a13b38 .config console log report info ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in __hrtimer_run_queues
2021/01/09 18:24 upstream 996e435fd401 2c1f2513 .config console log report info ci-upstream-kasan-gce-root
2021/01/02 15:40 upstream eda809aef534 79264ae3 .config console log report info ci-upstream-kasan-gce-root
2020/12/11 09:30 upstream 33dc9614dc20 f900b48c .config console log report info ci-upstream-kasan-gce-root
2020/12/11 07:50 upstream 33dc9614dc20 f900b48c .config console log report info ci-upstream-kasan-gce-root
2020/12/10 16:04 upstream a2f5ea9e314b f900b48c .config console log report info ci-upstream-kasan-gce-root
2020/12/09 18:45 upstream a68a0262abda c090b4da .config console log report info ci-upstream-kasan-gce-root
2020/12/09 14:58 upstream 7d8761ba27fc 40cc414d .config console log report info ci-upstream-kasan-gce-root
2020/12/06 18:47 upstream 7059c2c00a21 c521566d .config console log report info ci-upstream-kasan-gce-root
2020/12/05 03:22 upstream e87297fa080a 20366b87 .config console log report info ci-upstream-kasan-gce-root
2020/12/01 01:26 upstream b65054597872 b3a34598 .config console log report info ci-upstream-kasan-gce-root
2020/11/29 01:27 upstream 45e885c439e8 a0092f9d .config console log report info ci-upstream-kasan-gce-root
2020/11/17 05:02 net-old e2142ef266c8 1bf9a662 .config console log report info ci-upstream-net-this-kasan-gce
2020/10/14 18:09 net-old 1e40d75ef90c fc7735a2 .config console log report info ci-upstream-net-this-kasan-gce
* Struck through repros no longer work on HEAD.