INFO: rcu detected stall in sync_hw

Kernel	Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
linux-5.15	INFO: rcu detected stall in sync_hw_clock origin:upstream	C			2	27d	134d	0/3	upstream: reported C repro on 2024/01/04 12:38
upstream	BUG: soft lockup in sync_hw_clock efi	C			1	30d	26d	0/26	upstream: reported C repro on 2024/04/21 14:58

rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	1-...!: (1 GPs behind) idle=b5d4/1/0x4000000000000000 softirq=152401/152403 fqs=1
	(detected by 0, t=10506 jiffies, g=247629, q=68 ncpus=2)
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 600 Comm: kworker/1:13 Not tainted 6.1.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
Workqueue: events_freezable_power_ sync_hw_clock
RIP: 0010:__kasan_check_read+0x8/0x20 mm/kasan/shadow.c:31
Code: 0f 0b 48 83 c4 60 5b 5d 41 5c c3 48 05 00 80 00 00 48 89 fb 48 39 c7 0f 82 91 9d dd 07 eb e1 0f 1f 00 f3 0f 1e fa 48 8b 0c 24 <89> f6 31 d2 e9 8f f7 ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
RSP: 0018:ffffc900003e8bb0 EFLAGS: 00000002
RAX: 0000000000000001 RBX: 000000000000005a RCX: ffffffff8161d0b3
RDX: 1ffff11005760568 RSI: 0000000000000008 RDI: ffffffff910eba48
RBP: 0000000000000040 R08: 0000000000000000 R09: ffffffff910ebc2f
R10: fffffbfff221d785 R11: 0000000000000000 R12: ffff88802bb02b20
R13: ffff88802bb02080 R14: 0000000000000002 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005555557da848 CR3: 000000002abe0000 CR4: 00000000003506e0
Call Trace:
 <IRQ>
 instrument_atomic_read include/linux/instrumented.h:72 [inline]
 _test_bit include/asm-generic/bitops/instrumented-non-atomic.h:141 [inline]
 hlock_class kernel/locking/lockdep.c:227 [inline]
 __lock_acquire+0xbc3/0x56d0 kernel/locking/lockdep.c:5052
 lock_acquire kernel/locking/lockdep.c:5668 [inline]
 lock_acquire+0x1e3/0x630 kernel/locking/lockdep.c:5633
 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
 _raw_spin_lock_irq+0x36/0x50 kernel/locking/spinlock.c:170
 __run_hrtimer kernel/time/hrtimer.c:1689 [inline]
 __hrtimer_run_queues+0x266/0xfb0 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x320/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1096 [inline]
 __sysvec_apic_timer_interrupt+0x180/0x640 arch/x86/kernel/apic/apic.c:1113
 sysvec_apic_timer_interrupt+0x92/0xc0 arch/x86/kernel/apic/apic.c:1107
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x3c/0x70 kernel/locking/spinlock.c:194
Code: 74 24 10 e8 f6 fd 71 f7 48 89 ef e8 9e 6a 72 f7 81 e3 00 02 00 00 75 25 9c 58 f6 c4 02 75 2d 48 85 db 74 01 fb bf 01 00 00 00 <e8> cf f3 64 f7 65 8b 05 c8 55 12 76 85 c0 74 0a 5b 5d c3 e8 dc 97
RSP: 0018:ffffc900049c7be8 EFLAGS: 00000206
RAX: 0000000000000006 RBX: 0000000000000200 RCX: 1ffffffff2234936
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001
RBP: ffff8880b9b2b580 R08: 0000000000000001 R09: ffffffff910ebc2f
R10: 0000000000000001 R11: 0000000000000001 R12: 1908a608cd86db00
R13: 000000001dcd6500 R14: 000000006b84ef9e R15: 1908a608741eac00
 hrtimer_start include/linux/hrtimer.h:418 [inline]
 sched_sync_hw_clock kernel/time/ntp.c:519 [inline]
 sync_hw_clock+0x1fa/0x4d0 kernel/time/ntp.c:660
 process_one_work+0x9bf/0x1710 kernel/workqueue.c:2289
 worker_thread+0x669/0x1090 kernel/workqueue.c:2436
 kthread+0x2e8/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
 </TASK>
rcu: rcu_preempt kthread starved for 10500 jiffies! g247629 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:28408 pid:15    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5209 [inline]
 __schedule+0xae9/0x53f0 kernel/sched/core.c:6521
 schedule+0xde/0x1b0 kernel/sched/core.c:6597
 schedule_timeout+0x14e/0x2a0 kernel/time/timer.c:1935
 rcu_gp_fqs_loop+0x190/0x910 kernel/rcu/tree.c:1660
 rcu_gp_kthread+0x23a/0x360 kernel/rcu/tree.c:1859
 kthread+0x2e8/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
CPU: 0 PID: 11454 Comm: kworker/u4:0 Not tainted 6.1.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:413 [inline]
RIP: 0010:smp_call_function_many_cond+0x43f/0x10a0 kernel/smp.c:987
Code: e6 e8 95 f7 0a 00 45 85 e4 74 48 48 8b 04 24 49 89 c5 83 e0 07 49 c1 ed 03 49 89 c4 4d 01 fd 41 83 c4 03 e8 c3 fa 0a 00 f3 90 <41> 0f b6 45 00 41 38 c4 7c 08 84 c0 0f 85 4e 0a 00 00 8b 43 08 31
RSP: 0018:ffffc9000408f970 EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff8880b9b3f120 RCX: 0000000000000000
RDX: ffff88807a334040 RSI: ffffffff817440fd RDI: 0000000000000005
RBP: 0000000000000001 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000003
R13: ffffed1017367e25 R14: 0000000000000001 R15: dffffc0000000000
FS:  0000000000000000(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f824efabf84 CR3: 000000000c28e000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 on_each_cpu_cond_mask+0x5a/0xa0 kernel/smp.c:1155
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:1311 [inline]
 text_poke_bp_batch+0x3f1/0x6b0 arch/x86/kernel/alternative.c:1555
 text_poke_flush arch/x86/kernel/alternative.c:1670 [inline]
 text_poke_flush arch/x86/kernel/alternative.c:1667 [inline]
 text_poke_finish+0x1a/0x30 arch/x86/kernel/alternative.c:1677
 arch_jump_label_transform_apply+0x17/0x30 arch/x86/kernel/jump_label.c:146
 jump_label_update+0x32f/0x410 kernel/jump_label.c:801
 static_key_enable_cpuslocked+0x1b5/0x270 kernel/jump_label.c:177
 static_key_enable+0x1a/0x20 kernel/jump_label.c:190
 toggle_allocation_gate mm/kfence/core.c:800 [inline]
 toggle_allocation_gate+0x104/0x390 mm/kfence/core.c:792
 process_one_work+0x9bf/0x1710 kernel/workqueue.c:2289
 worker_thread+0x669/0x1090 kernel/workqueue.c:2436
 kthread+0x2e8/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
 </TASK>

Time	Kernel	Commit	Syzkaller	Config	Log	Report	VM info	Assets (help?)	Manager	Title
2022/11/28 17:29	upstream	b7b275e60bcd	247de55b	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-selinux-root	INFO: rcu detected stall in sync_hw_clock
2022/11/26 22:05	upstream	644e9524388a	74a66371	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-selinux-root	INFO: rcu detected stall in sync_hw_clock
2023/01/14 10:16	linux-next	0a093b2893c7	529798b0	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-linux-next-kasan-gce-root	INFO: rcu detected stall in sync_hw_clock