INFO: rcu detected stall in __hrtimer_run

INFO: rcu detected stall in __hrtimer_run_queues
Status: upstream: reported C repro on 2021/02/20 21:05
Reported-by: syzbot+de9526ade17c659d8336@syzkaller.appspotmail.com
First crash: 207d, last: 14d

Cause bisection: the issue happens on the oldest tested release (bisect log)
Crash: INFO: rcu detected stall in mac80211_hwsim_beacon (log)
Repro: C syz .config

Sample crash report:

rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-....: (10153 ticks this GP) idle=f1a/1/0x4000000000000000 softirq=10867/10868 fqs=925 
	(t=10502 jiffies g=10029 q=19103)
NMI backtrace for cpu 1
CPU: 1 PID: 10530 Comm: syz-executor248 Not tainted 5.11.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x107/0x163 lib/dump_stack.c:120
 nmi_cpu_backtrace.cold+0x44/0xd7 lib/nmi_backtrace.c:105
 nmi_trigger_cpumask_backtrace+0x1b3/0x230 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x1f4/0x230 kernel/rcu/tree_stall.h:337
 print_cpu_stall kernel/rcu/tree_stall.h:569 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:643 [inline]
 rcu_pending kernel/rcu/tree.c:3751 [inline]
 rcu_sched_clock_irq.cold+0x48e/0xedf kernel/rcu/tree.c:2580
 update_process_times+0x16d/0x200 kernel/time/timer.c:1782
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1369
 __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
 __hrtimer_run_queues+0x1c0/0xe40 kernel/time/hrtimer.c:1583
 hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
 run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:91 [inline]
 sysvec_apic_timer_interrupt+0x48/0x100 arch/x86/kernel/apic/apic.c:1100
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:629
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:161 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x50 kernel/locking/spinlock.c:191
Code: f8 5d c3 66 90 55 48 89 fd 48 83 c7 18 53 48 89 f3 48 8b 74 24 10 e8 9a d6 5b f8 48 89 ef e8 42 8b 5c f8 f6 c7 02 75 1a 53 9d <bf> 01 00 00 00 e8 81 92 50 f8 65 8b 05 7a f8 04 77 85 c0 74 0a 5b
RSP: 0018:ffffc90000db0e48 EFLAGS: 00000286
RAX: 0000000000ce281a RBX: 0000000000000286 RCX: 1ffffffff1b46a19
RDX: 0000000000000000 RSI: 0000000000000102 RDI: 0000000000000000
RBP: ffff8880b9d26a00 R08: 0000000000000001 R09: 0000000000000001
R10: ffffffff8178a8b8 R11: 0000000000000000 R12: 000000a80fcc296a
R13: ffff8880b9d26c80 R14: ffff8880b9d26a00 R15: ffffffff851589e0
 __run_hrtimer kernel/time/hrtimer.c:1515 [inline]
 __hrtimer_run_queues+0x51a/0xe40 kernel/time/hrtimer.c:1583
 hrtimer_run_softirq+0x17b/0x360 kernel/time/hrtimer.c:1600
 __do_softirq+0x29b/0x9f6 kernel/softirq.c:343
 asm_call_irq_on_stack+0xf/0x20
 </IRQ>
 __run_on_irqstack arch/x86/include/asm/irq_stack.h:26 [inline]
 run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:77 [inline]
 do_softirq_own_stack+0xaa/0xd0 arch/x86/kernel/irq_64.c:77
 invoke_softirq kernel/softirq.c:226 [inline]
 __irq_exit_rcu kernel/softirq.c:420 [inline]
 irq_exit_rcu+0x134/0x200 kernel/softirq.c:432
 sysvec_apic_timer_interrupt+0x4d/0x100 arch/x86/kernel/apic/apic.c:1100
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:629
RIP: 0010:queue_work_on+0x83/0xd0 kernel/workqueue.c:1530
Code: 31 ff 89 ee e8 6e 02 29 00 40 84 ed 74 46 e8 e4 fb 28 00 31 ff 48 89 de e8 ca 03 29 00 48 85 db 75 26 e8 d0 fb 28 00 41 56 9d <48> 83 c4 08 44 89 f8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 e8 b6 fb 28
RSP: 0018:ffffc90002e5fc80 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 0000000000000200 RCX: 0000000000000000
RDX: ffff8880265f5340 RSI: ffffffff8149da20 RDI: 0000000000000000
RBP: ffffc90002e68000 R08: 0000000000000001 R09: 0000000000000001
R10: ffffffff8178a8b8 R11: 0000000000000000 R12: ffff8880b9d31568
R13: ffff888010c64c00 R14: 0000000000000293 R15: 0000000000000001
 queue_work include/linux/workqueue.h:507 [inline]
 schedule_work include/linux/workqueue.h:568 [inline]
 __vfree_deferred mm/vmalloc.c:2307 [inline]
 vfree_atomic+0xac/0xe0 mm/vmalloc.c:2325
 free_thread_stack kernel/fork.c:291 [inline]
 release_task_stack kernel/fork.c:428 [inline]
 put_task_stack+0x29c/0x480 kernel/fork.c:439
 finish_task_switch.isra.0+0x557/0x7e0 kernel/sched/core.c:4236
 context_switch kernel/sched/core.c:4330 [inline]
 __schedule+0x914/0x21a0 kernel/sched/core.c:5078
 preempt_schedule_common+0x45/0xc0 kernel/sched/core.c:5238
 preempt_schedule_thunk+0x16/0x18 arch/x86/entry/thunk_64.S:35
 __raw_spin_unlock include/linux/spinlock_api_smp.h:152 [inline]
 _raw_spin_unlock+0x36/0x40 kernel/locking/spinlock.c:183
 spin_unlock include/linux/spinlock.h:394 [inline]
 pick_file+0x129/0x1e0 fs/file.c:613
 close_fd+0x44/0x80 fs/file.c:622
 __do_sys_close fs/open.c:1299 [inline]
 __se_sys_close fs/open.c:1297 [inline]
 __x64_sys_close+0x2f/0xa0 fs/open.c:1297
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x403353
Code: c7 c2 c0 ff ff ff f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8
RSP: 002b:00007ffdb38739a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 0000000000403353
RDX: 0000000000042000 RSI: 0000000000000004 RDI: 0000000000000003
RBP: 0000000000000000 R08: 0000000000000031 R09: 0000000000000031
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000000aa47b
R13: 00007ffdb3873a00 R14: 00007ffdb38739f0 R15: 00007ffdb38739c4

Fix bisection attempts:
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	VM info	Title
ci-upstream-kasan-gce-root	2021/03/25 20:20	upstream	00232240	98682e5e	.config	log	report	syz	C

Crashes (19):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	VM info	Title
ci-upstream-kasan-gce-root	2021/02/16 20:47	upstream	f40ddce8	98682e5e	.config	log	report	syz	C		INFO: rcu detected stall in __hrtimer_run_queues
ci-upstream-kasan-gce-root	2021/04/24 20:12	upstream	8db5efb8	17f0b706	.config	log	report			info	INFO: rcu detected stall in __hrtimer_run_queues
ci-upstream-kasan-gce-root	2021/02/23 19:51	upstream	3b9cdafb	fcc6d71b	.config	log	report			info	INFO: rcu detected stall in __hrtimer_run_queues
ci-upstream-kasan-gce-root	2021/02/22 09:19	upstream	31caf8b2	a659b3f1	.config	log	report			info	INFO: rcu detected stall in __hrtimer_run_queues
ci-upstream-kasan-gce-root	2021/02/22 08:41	upstream	31caf8b2	a659b3f1	.config	log	report			info	INFO: rcu detected stall in __hrtimer_run_queues
ci-upstream-kasan-gce-root	2021/01/29 12:14	upstream	bec4c296	6593fd32	.config	log	report			info	INFO: rcu detected stall in __hrtimer_run_queues
ci-upstream-kasan-gce-root	2021/01/09 18:24	upstream	996e435f	2c1f2513	.config	log	report			info
ci-upstream-kasan-gce-root	2021/01/02 15:40	upstream	eda809ae	79264ae3	.config	log	report			info
ci-upstream-kasan-gce-root	2020/12/11 09:30	upstream	33dc9614	f900b48c	.config	log	report			info
ci-upstream-kasan-gce-root	2020/12/11 07:50	upstream	33dc9614	f900b48c	.config	log	report			info
ci-upstream-kasan-gce-root	2020/12/10 16:04	upstream	a2f5ea9e	f900b48c	.config	log	report			info
ci-upstream-kasan-gce-root	2020/12/09 18:45	upstream	a68a0262	c090b4da	.config	log	report			info
ci-upstream-kasan-gce-root	2020/12/09 14:58	upstream	7d8761ba	40cc414d	.config	log	report			info
ci-upstream-kasan-gce-root	2020/12/06 18:47	upstream	7059c2c0	c521566d	.config	log	report			info
ci-upstream-kasan-gce-root	2020/12/05 03:22	upstream	e87297fa	20366b87	.config	log	report			info
ci-upstream-kasan-gce-root	2020/12/01 01:26	upstream	b6505459	b3a34598	.config	log	report			info
ci-upstream-kasan-gce-root	2020/11/29 01:27	upstream	45e885c4	a0092f9d	.config	log	report			info
ci-upstream-net-this-kasan-gce	2020/11/17 05:02	net	e2142ef2	1bf9a662	.config	log	report			info
ci-upstream-net-this-kasan-gce	2020/10/14 18:09	net	1e40d75e	fc7735a2	.config	log	report			info