syzbot


INFO: rcu detected stall in worker_thread (2)

Status: upstream: reported on 2024/06/10 22:16
Reported-by: syzbot+a8b639ddec1e095f1806@syzkaller.appspotmail.com
First crash: 8d21h, last: 3d14h
Similar bugs (11)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in worker_thread (3) cgroups mm 1 1624d 1624d 0/27 closed as invalid on 2020/01/08 05:33
upstream INFO: rcu detected stall in worker_thread (4) cgroups mm 28 1624d 1624d 0/27 closed as invalid on 2020/01/09 08:13
linux-5.15 INFO: rcu detected stall in worker_thread 1 264d 264d 0/3 auto-obsoleted due to no activity on 2024/01/09 18:17
upstream INFO: rcu detected stall in worker_thread (8) kernel 1 407d 407d 0/27 auto-obsoleted due to no activity on 2023/08/07 05:14
upstream INFO: rcu detected stall in worker_thread (5) kernel 2 748d 785d 0/27 auto-closed as invalid on 2022/08/31 00:50
upstream INFO: rcu detected stall in worker_thread (9) mm C done 768 2h12m 256d 0/27 upstream: reported C repro on 2023/10/07 18:33
upstream INFO: rcu detected stall in worker_thread (2) cgroups mm 12 1624d 1624d 0/27 closed as invalid on 2020/01/08 05:23
upstream INFO: rcu detected stall in worker_thread cgroups mm 150 1659d 1660d 0/27 closed as invalid on 2019/12/04 14:14
linux-6.1 INFO: rcu detected stall in worker_thread 6 2h44m 30d 0/3 upstream: reported on 2024/05/20 10:24
upstream INFO: rcu detected stall in worker_thread (6) kernel 1 624d 624d 0/27 auto-obsoleted due to no activity on 2023/01/12 15:50
upstream INFO: rcu detected stall in worker_thread (7) kernel 1 517d 517d 0/27 auto-obsoleted due to no activity on 2023/04/27 16:37

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-...!: (10876 ticks this GP) idle=fad/1/0x4000000000000000 softirq=8973/8976 fqs=1 
	(t=10500 jiffies g=9529 q=37)
rcu: rcu_preempt kthread starved for 10490 jiffies! g9529 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:26368 pid:   15 ppid:     2 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5030 [inline]
 __schedule+0x12c4/0x45b0 kernel/sched/core.c:6376
 schedule+0x11b/0x1f0 kernel/sched/core.c:6459
 schedule_timeout+0x1b9/0x300 kernel/time/timer.c:1914
 rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
 rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 4469 Comm: syz-executor.1 Not tainted 5.15.160-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/07/2024
RIP: 0010:check_preemption_disabled+0x5c/0x110 lib/smp_processor_id.c:19
Code: 25 28 00 00 00 48 3b 44 24 08 0f 85 c7 00 00 00 89 d8 48 83 c4 10 5b 41 5c 41 5e 41 5f c3 48 c7 04 24 00 00 00 00 9c 8f 04 24 <f7> 04 24 00 02 00 00 74 c9 49 89 f6 49 89 ff 65 4c 8b 25 dd 6d e4
RSP: 0018:ffffc900033d7e90 EFLAGS: 00000046
RAX: 0000000080000000 RBX: 0000000000000000 RCX: 0000000000040000
RDX: ffffc90005d79000 RSI: ffffffff8a8b2980 RDI: ffffffff8ad8f6c0
RBP: ffffc900033d7f48 R08: dffffc0000000000 R09: fffffbfff1bc8cce
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000000000000
R13: 0000000000000000 R14: ffffffff8a1dfba5 R15: 0000000000000000
FS:  00007fc43cb5d6c0(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b2fd52000 CR3: 000000001dee7000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 lockdep_hardirqs_off+0x70/0x100 kernel/locking/lockdep.c:4379
 trace_hardirqs_off+0x14/0x70 kernel/trace/trace_preemptirq.c:76
 local_irq_disable_exit_to_user include/linux/entry-common.h:211 [inline]
 __syscall_exit_to_user_mode_work kernel/entry/common.c:295 [inline]
 syscall_exit_to_user_mode+0x55/0x240 kernel/entry/common.c:307
 do_syscall_64+0x47/0xb0 arch/x86/entry/common.c:86
 entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7fc43e002627
Code: 0b e9 68 fe ff ff 48 83 c4 18 48 8d 3d 32 02 c7 00 5b 5d 41 5c 41 5d 41 5e 41 5f e9 c3 68 fd ff 0f 1f 00 b8 27 00 00 00 0f 05 <c3> 0f 1f 84 00 00 00 00 00 b8 66 00 00 00 0f 05 c3 0f 1f 84 00 00
RSP: 002b:00007fc43cb5cb88 EFLAGS: 00000206 ORIG_RAX: 0000000000000027
RAX: 00000000000000a4 RBX: 00007fc43cb5ccf0 RCX: 00007fc43e002627
RDX: 00007fc43cb5cbc0 RSI: 00007fc43cb5ccf0 RDI: 0000000000000021
RBP: 0000000000000000 R08: 00007fc43cb5d6c0 R09: 00007fc43cb5d6c0
R10: 0000000000000000 R11: 0000000000000206 R12: 00007fc43e13c05c
R13: 000000000000006e R14: 00007ffecba1a760 R15: 00007ffecba1a848
 </TASK>
NMI backtrace for cpu 1
CPU: 1 PID: 1066 Comm: kworker/1:2 Not tainted 5.15.160-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/07/2024
Workqueue:  0x0 (events)
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
 nmi_cpu_backtrace+0x46a/0x4a0 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x181/0x2a0 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:166 [inline]
 rcu_dump_cpu_stacks+0x223/0x390 kernel/rcu/tree_stall.h:349
 print_cpu_stall+0x320/0x600 kernel/rcu/tree_stall.h:633
 check_cpu_stall kernel/rcu/tree_stall.h:727 [inline]
 rcu_pending kernel/rcu/tree.c:3932 [inline]
 rcu_sched_clock_irq+0x8d9/0x1150 kernel/rcu/tree.c:2619
 update_process_times+0x196/0x200 kernel/time/timer.c:1818
 tick_sched_handle kernel/time/tick-sched.c:254 [inline]
 tick_sched_timer+0x386/0x550 kernel/time/tick-sched.c:1473
 __run_hrtimer kernel/time/hrtimer.c:1686 [inline]
 __hrtimer_run_queues+0x55b/0xcf0 kernel/time/hrtimer.c:1750
 hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1812
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1085 [inline]
 __sysvec_apic_timer_interrupt+0x139/0x470 arch/x86/kernel/apic/apic.c:1102
 sysvec_apic_timer_interrupt+0x8c/0xb0 arch/x86/kernel/apic/apic.c:1096
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:finish_lock_switch+0x91/0x100 kernel/sched/core.c:4785
Code: 45 31 c9 68 b7 90 59 81 e8 cc 1a 09 00 48 83 c4 08 4c 89 ff e8 60 da fe ff eb 32 4c 89 ff e8 f6 da cd 08 e8 d1 4b 2d 00 fb 5b <41> 5c 41 5d 41 5e 41 5f c3 44 89 f1 80 e1 07 80 c1 03 38 c1 7c 87
RSP: 0018:ffffc9000484fb38 EFLAGS: 00000286
RAX: 760503f8f12dd100 RBX: ffff88807d0b8034 RCX: ffffffff913eff03
RDX: dffffc0000000000 RSI: ffffffff8a8b2980 RDI: ffffffff8ad8f6c0
RBP: ffffc9000484fbb0 R08: ffffffff8186dcf0 R09: ffffed1017347469
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: 1ffff11017367613 R14: ffff8880b9b3b098 R15: ffff8880b9a3a340
 finish_task_switch+0x134/0x630 kernel/sched/core.c:4902
 context_switch kernel/sched/core.c:5033 [inline]
 __schedule+0x12cc/0x45b0 kernel/sched/core.c:6376
 schedule+0x11b/0x1f0 kernel/sched/core.c:6459
 worker_thread+0xf56/0x1280 kernel/workqueue.c:2478
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300
 </TASK>

Crashes (4):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/06/16 04:33 linux-5.15.y c61bd26ae81a f429ab00 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in worker_thread
2024/06/13 15:06 linux-5.15.y c61bd26ae81a a9616ff5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in worker_thread
2024/06/11 02:00 linux-5.15.y c61bd26ae81a 048c640a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in worker_thread
2024/06/10 22:15 linux-5.15.y c61bd26ae81a 048c640a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in worker_thread
* Struck through repros no longer work on HEAD.