syzbot


INFO: rcu detected stall in cleanup_mnt (2)

Status: upstream: reported C repro on 2024/08/10 23:43
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+09e7336b5c7f5b7fa856@syzkaller.appspotmail.com
First crash: 188d, last: 23d
Fix commit to backport (bisect log) :
tree: upstream
commit e634134180885574d1fe7aa162777ba41e7fcd5b
Author: Vladimir Oltean <vladimir.oltean@nxp.com>
Date: Mon May 27 15:39:54 2024 +0000

  net/sched: taprio: make q->picos_per_byte available to fill_sched_entry()

  
Bug presence (2)
Date Name Commit Repro Result
2025/01/01 linux-5.15.y (ToT) 91786f140358 C [report] INFO: rcu detected stall in cleanup_mnt
2025/01/01 upstream (ToT) ccb98ccef0e5 C Didn't crash
Similar bugs (6)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 INFO: rcu detected stall in cleanup_mnt 1 615d 615d 0/3 auto-obsoleted due to no activity on 2023/09/19 04:00
upstream INFO: rcu detected stall in cleanup_mnt (2) block 1 1151d 1151d 0/28 closed as invalid on 2022/02/08 10:10
linux-6.1 INFO: rcu detected stall in cleanup_mnt (2) 1 244d 244d 0/3 auto-obsoleted due to no activity on 2024/09/23 19:34
linux-6.1 INFO: rcu detected stall in cleanup_mnt 1 560d 560d 0/3 auto-obsoleted due to no activity on 2023/11/13 00:06
upstream INFO: rcu detected stall in cleanup_mnt exfat 1 1682d 1682d 0/28 auto-closed as invalid on 2020/09/06 14:28
upstream INFO: rcu detected stall in cleanup_mnt (3) reiserfs block 2 626d 636d 0/28 auto-obsoleted due to no activity on 2023/08/29 03:36
Last patch testing requests (2)
Created Duration User Patch Repo Result
2025/01/19 11:48 11m retest repro linux-5.15.y report log
2024/11/10 11:07 16m retest repro linux-5.15.y report log
Fix bisection attempts (1)
Created Duration User Patch Repo Result
2024/09/30 14:24 8h35m fix candidate upstream OK (1) job log

Sample crash report:
sched: RT throttling activated
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
	(detected by 0, t=10502 jiffies, g=5501, q=86)
rcu: All QSes seen, last rcu_preempt kthread activity 10503 (4294967905-4294957402), jiffies_till_next_fqs=1, root ->qsmask 0x0
rcu: rcu_preempt kthread starved for 10504 jiffies! g5501 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:27064 pid:   15 ppid:     2 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5027 [inline]
 __schedule+0x12c4/0x45b0 kernel/sched/core.c:6373
 schedule+0x11b/0x1f0 kernel/sched/core.c:6456
 schedule_timeout+0x1b9/0x300 kernel/time/timer.c:1914
 rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
 rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
NMI backtrace for cpu 0
CPU: 0 PID: 4188 Comm: syz-executor549 Not tainted 5.15.175-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
 nmi_cpu_backtrace+0x46a/0x4a0 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x181/0x2a0 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:166 [inline]
 rcu_check_gp_kthread_starvation+0x1d2/0x240 kernel/rcu/tree_stall.h:487
 print_other_cpu_stall+0x137a/0x14d0 kernel/rcu/tree_stall.h:592
 check_cpu_stall kernel/rcu/tree_stall.h:745 [inline]
 rcu_pending kernel/rcu/tree.c:3932 [inline]
 rcu_sched_clock_irq+0xa38/0x1150 kernel/rcu/tree.c:2619
 update_process_times+0x196/0x200 kernel/time/timer.c:1818
 tick_sched_handle kernel/time/tick-sched.c:254 [inline]
 tick_sched_timer+0x386/0x550 kernel/time/tick-sched.c:1473
 __run_hrtimer kernel/time/hrtimer.c:1688 [inline]
 __hrtimer_run_queues+0x55b/0xcf0 kernel/time/hrtimer.c:1752
 hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1814
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1097 [inline]
 __sysvec_apic_timer_interrupt+0x13b/0x4b0 arch/x86/kernel/apic/apic.c:1114
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1108 [inline]
 sysvec_apic_timer_interrupt+0x9b/0xc0 arch/x86/kernel/apic/apic.c:1108
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:676
RIP: 0010:radix_tree_next_chunk+0x64f/0xb30 lib/radix-tree.c:1216
Code: 78 14 8d 4c 89 fe 48 89 eb 48 89 ea e8 7a d3 fd ff 48 89 e8 49 bc 00 00 00 00 00 fc ff df e9 e2 fd ff ff e8 43 cc 48 fd eb 05 <e8> 3c cc 48 fd 48 8b 44 24 50 42 80 3c 20 00 4c 8b 6c 24 30 74 08
RSP: 0018:ffffc90002eaf768 EFLAGS: 00000246
RAX: 0000000000000000 RBX: 0000000000001000 RCX: ffff888022a89dc0
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000040
RBP: 0000000000000000 R08: ffffffff8437b6b8 R09: ffffed102810261e
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: ffff888140812ec0 R14: 0000000000000006 R15: 0000000000000040
 radix_tree_gang_lookup_tag+0x18e/0x430 lib/radix-tree.c:1311
 xfs_icwalk_ag+0x274/0x1a10 fs/xfs/xfs_icache.c:1680
 xfs_icwalk fs/xfs/xfs_icache.c:1773 [inline]
 xfs_reclaim_inodes+0x1f3/0x300 fs/xfs/xfs_icache.c:993
 xfs_unmount_flush_inodes fs/xfs/xfs_mount.c:554 [inline]
 xfs_unmountfs+0x14f/0x270 fs/xfs/xfs_mount.c:1016
 xfs_fs_put_super+0x65/0x2b0 fs/xfs/xfs_super.c:1096
 generic_shutdown_super+0x130/0x310 fs/super.c:475
 kill_block_super+0x7a/0xe0 fs/super.c:1427
 deactivate_locked_super+0xa0/0x110 fs/super.c:335
 cleanup_mnt+0x44e/0x500 fs/namespace.c:1143
 task_work_run+0x129/0x1a0 kernel/task_work.c:188
 tracehook_notify_resume include/linux/tracehook.h:189 [inline]
 exit_to_user_mode_loop+0x106/0x130 kernel/entry/common.c:181
 exit_to_user_mode_prepare+0xb1/0x140 kernel/entry/common.c:214
 __syscall_exit_to_user_mode_work kernel/entry/common.c:296 [inline]
 syscall_exit_to_user_mode+0x5d/0x240 kernel/entry/common.c:307
 do_syscall_64+0x47/0xb0 arch/x86/entry/common.c:86
 entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7fb81e0d5eda
Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb a6 e8 5e 04 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fff1e670068 EFLAGS: 00000242 ORIG_RAX: 00000000000000a5
RAX: ffffffffffffffec RBX: 00007fff1e670080 RCX: 00007fb81e0d5eda
RDX: 0000000020009800 RSI: 0000000020000140 RDI: 00007fff1e670080
RBP: 0000000000000004 R08: 00007fff1e6700c0 R09: 0000000000009855
R10: 0000000001000000 R11: 0000000000000242 R12: 0000000001000000
R13: 00007fff1e6700c0 R14: 0000000000000003 R15: 0000000001000000
 </TASK>

Crashes (7):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/12/31 07:43 linux-5.15.y 91786f140358 d3ccff63 .config console log report syz / log C [disk image] [vmlinux] [kernel image] [mounted in repro] ci2-linux-5-15-kasan INFO: rcu detected stall in cleanup_mnt
2024/09/01 02:08 linux-5.15.y fa93fa65db6e 1eda0d14 .config console log report syz / log [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in cleanup_mnt
2025/01/22 19:12 linux-5.15.y 4735586da88e a44b0418 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in cleanup_mnt
2024/12/30 22:33 linux-5.15.y 91786f140358 d3ccff63 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in cleanup_mnt
2024/10/21 14:05 linux-5.15.y 584a40a22cb9 f1e4447c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in cleanup_mnt
2024/09/23 11:34 linux-5.15.y 3a5928702e71 89298aad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in cleanup_mnt
2024/08/10 23:42 linux-5.15.y 7e89efd3ae1c 6f4edef4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in cleanup_mnt
* Struck through repros no longer work on HEAD.