INFO: rcu detected stall in sync_hw_clock

Status: upstream: reported C repro on 2024/01/04 12:38
Bug presence: origin:upstream
[Documentation on labels]
Reported-by: syzbot+7d6bde386482547466a4@syzkaller.appspotmail.com
First crash: 931d, last: 189d
Fix bisection: failed (error log, bisect log)
  
▶ ▼ Bug presence (1)
Date Name Commit Repro Result
2024/01/04 upstream (ToT) ac865f00af29 C [report] INFO: rcu detected stall in worker_thread
▶ ▼ Similar bugs (4)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in sync_hw_clock (2) rtc 1 2 615d 626d 0/29 auto-obsoleted due to no activity on 2025/02/13 00:09
upstream INFO: rcu detected stall in sync_hw_clock kernel 1 3 1286d 1334d 0/29 auto-obsoleted due to no activity on 2023/04/21 23:58
linux-6.1 INFO: rcu detected stall in sync_hw_clock 1 2 556d 559d 0/3 auto-obsoleted due to no activity on 2025/04/23 12:58
upstream BUG: soft lockup in sync_hw_clock efi 1 C 1 827d 823d 0/29 auto-obsoleted due to no activity on 2024/07/26 14:56
▶ ▼ Last patch testing requests (2)
Created Duration User Patch Repo Result
2026/07/10 22:27 17m retest repro linux-5.15.y report log
2026/04/28 06:45 47m retest repro linux-5.15.y report log
▶ ▼ Fix bisection attempts (22)
Created Duration User Patch Repo Result
2026/02/14 10:23 0m bisect fix linux-5.15.y error job log
2026/01/15 08:09 2h12m bisect fix linux-5.15.y OK (0) job log log
2025/12/12 17:16 1h52m bisect fix linux-5.15.y OK (0) job log log
2025/11/08 21:09 2h05m bisect fix linux-5.15.y OK (0) job log log
2025/10/01 15:15 1h31m bisect fix linux-5.15.y OK (0) job log log
2025/08/25 09:40 3h19m bisect fix linux-5.15.y OK (0) job log log
2025/07/22 14:08 3h52m bisect fix linux-5.15.y OK (0) job log log
2025/06/20 15:06 2h07m bisect fix linux-5.15.y OK (0) job log log
2025/05/21 04:57 1h45m bisect fix linux-5.15.y OK (0) job log log
2025/04/17 21:27 4h44m bisect fix linux-5.15.y OK (0) job log log
2025/03/15 07:33 1h47m bisect fix linux-5.15.y OK (0) job log log
2025/02/11 17:59 2h17m bisect fix linux-5.15.y OK (0) job log log
2025/01/09 15:11 2h05m bisect fix linux-5.15.y OK (0) job log log
2024/11/19 14:40 1h48m bisect fix linux-5.15.y OK (0) job log log
2024/10/16 04:44 2h25m bisect fix linux-5.15.y OK (0) job log log
2024/09/16 00:54 2h27m bisect fix linux-5.15.y OK (0) job log log
2024/08/13 09:58 3h06m bisect fix linux-5.15.y OK (0) job log log
2024/07/01 23:52 2h01m bisect fix linux-5.15.y OK (0) job log log
2024/05/25 05:25 1h13m bisect fix linux-5.15.y OK (0) job log log
2024/04/20 17:46 1h44m bisect fix linux-5.15.y OK (0) job log log
2024/03/18 09:33 1h10m bisect fix linux-5.15.y OK (0) job log log
2024/02/11 05:33 1h09m bisect fix linux-5.15.y OK (0) job log log

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
	(detected by 1, t=10502 jiffies, g=3689, q=151)
rcu: All QSes seen, last rcu_preempt kthread activity 10502 (4294966185-4294955683), jiffies_till_next_fqs=1, root ->qsmask 0x0
rcu: rcu_preempt kthread starved for 10502 jiffies! g3689 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:27000 pid:   15 ppid:     2 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5030 [inline]
 __schedule+0x12c4/0x45b0 kernel/sched/core.c:6376
 schedule+0x11b/0x1f0 kernel/sched/core.c:6459
 schedule_timeout+0x1b9/0x300 kernel/time/timer.c:1884
 rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
 rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
 kthread+0x3f6/0x4f0 kernel/kthread.c:319
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 13 Comm: kworker/0:1 Not tainted 5.15.145-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/17/2023
Workqueue: events_freezable_power_ sync_hw_clock
RIP: 0010:lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
Code: 41 0f 94 c1 48 8b 7c 24 20 44 89 f6 8b 54 24 1c 48 8b 4c 24 30 4c 8b 44 24 38 6a 00 6a 00 ff 75 10 ff 74 24 40 e8 55 04 00 00 <48> 83 c4 20 48 c7 c7 20 22 8b 8a e8 b5 8d b9 08 b8 ff ff ff ff 65
RSP: 0018:ffffc90000007ac0 EFLAGS: 00000086
RAX: 0000000000000001 RBX: ffffc90000007b40 RCX: ffffffff81633742
RDX: dffffc0000000000 RSI: 0000000000000001 RDI: ffffffff8fbcf0c0
RBP: ffffc90000007c28 R08: dffffc0000000000 R09: fffffbfff1f79e19
R10: 0000000000000000 R11: dffffc0000000001 R12: 1ffff92000000f64
R13: dffffc0000000000 R14: 0000000000000000 R15: 0000000000000046
FS:  0000000000000000(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020000600 CR3: 000000001d32a000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
 _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
 debug_object_deactivate+0x61/0x3f0 lib/debugobjects.c:759
 debug_hrtimer_deactivate kernel/time/hrtimer.c:425 [inline]
 debug_deactivate+0x1b/0x1e0 kernel/time/hrtimer.c:481
 __run_hrtimer kernel/time/hrtimer.c:1653 [inline]
 __hrtimer_run_queues+0x31d/0xcf0 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1085 [inline]
 __sysvec_apic_timer_interrupt+0x139/0x470 arch/x86/kernel/apic/apic.c:1102
 sysvec_apic_timer_interrupt+0x8c/0xb0 arch/x86/kernel/apic/apic.c:1096
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:raw_spin_rq_unlock_irq+0x19/0x80 kernel/sched/sched.h:1339
Code: 38 c1 7c cb 4c 89 f7 e8 25 79 70 00 eb c1 0f 1f 00 41 57 41 56 53 48 89 fb 66 90 48 89 df e8 7e 14 cc 08 e8 59 3e 2d 00 fb 5b <41> 5e 41 5f c3 49 bf 00 00 00 00 00 fc ff df 4c 8d b3 58 0d 00 00
RSP: 0018:ffffc90000d27788 EFLAGS: 00000282
RAX: 9e7f07506b184b00 RBX: ffff8880b9a39700 RCX: ffffffff8162eb78
RDX: dffffc0000000000 RSI: ffffffff8a8b0f80 RDI: ffffffff8ad88080
RBP: ffffc90000d27970 R08: dffffc0000000000 R09: fffffbfff1f79e35
R10: 0000000000000000 R11: dffffc0000000001 R12: ffff88813fe20000
R13: dffffc0000000000 R14: ffff88813fe20000 R15: ffff88813fe20080
 __schedule+0x1462/0x45b0 kernel/sched/core.c:6382
 preempt_schedule_common+0x83/0xd0 kernel/sched/core.c:6552
 preempt_schedule+0xd9/0xe0 kernel/sched/core.c:6577
 preempt_schedule_thunk+0x16/0x18 arch/x86/entry/thunk_64.S:34
 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:161 [inline]
 _raw_spin_unlock_irqrestore+0x128/0x130 kernel/locking/spinlock.c:194
 hrtimer_start include/linux/hrtimer.h:418 [inline]
 sched_sync_hw_clock kernel/time/ntp.c:519 [inline]
 sync_hw_clock+0x418/0x520 kernel/time/ntp.c:660
 process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
 worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
 kthread+0x3f6/0x4f0 kernel/kthread.c:319
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
 </TASK>
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.351 msecs

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/01/04 12:38 linux-5.15.y d93fa2c78854 28c42cff .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in sync_hw_clock
2024/02/15 17:25 linux-5.15.y 6139f2a02fe0 fd39cf6f .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in sync_hw_clock
* Struck through repros no longer work on HEAD.