BUG: soft lockup in wb_workfn

Status: upstream: reported C repro on 2024/10/24 21:18
Bug presence: origin:upstream
[Documentation on labels]
Reported-by: syzbot+48cac2bbba146c43df3d@syzkaller.appspotmail.com
First crash: 637d, last: 39d
Fix bisection: failed (error log, bisect log)
  
▶ ▼ Bug presence (1)
Date Name Commit Repro Result
2025/05/31 upstream (ToT) dee264c16a63 C [report] INFO: rcu detected stall in worker_thread
▶ ▼ Similar bugs (9)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 INFO: rcu detected stall in wb_workfn origin:lts-only 1 syz error 4 652d 724d 0/3 auto-obsoleted due to no activity on 2025/01/21 00:06
upstream INFO: rcu detected stall in wb_workfn (4) prio:normal ext4 1 C error error 65 40d 377d 0/29 upstream: reported C repro on 2025/07/11 13:28
linux-5.15 INFO: rcu detected stall in wb_workfn (2) origin:lts-only 1 C error 106 23d 525d 0/3 upstream: reported C repro on 2025/02/13 22:15
upstream BUG: soft lockup in wb_workfn kernel 1 1 2536d 2532d 0/29 auto-closed as invalid on 2019/11/11 12:45
upstream INFO: rcu detected stall in wb_workfn (2) fs 1 1 1487d 1487d 0/29 auto-closed as invalid on 2022/09/25 22:53
upstream INFO: rcu detected stall in wb_workfn mm 1 2 1730d 1737d 0/29 auto-closed as invalid on 2022/01/25 21:50
linux-4.19 INFO: rcu detected stall in wb_workfn 1 1 1435d 1435d 0/1 auto-obsoleted due to no activity on 2022/12/16 09:39
upstream INFO: rcu detected stall in wb_workfn (3) ext4 block hfs 1 3 1068d 1150d 0/29 auto-obsoleted due to no activity on 2023/11/18 21:40
linux-4.14 INFO: rcu detected stall in wb_workfn 1 1 2468d 2468d 0/1 auto-closed as invalid on 2020/02/17 08:22
▶ ▼ Fix bisection attempts (3)
Created Duration User Patch Repo Result
2025/08/05 13:46 2h04m bisect fix linux-6.1.y error job log
2025/07/05 22:27 2h59m bisect fix linux-6.1.y OK (0) job log log
2025/02/23 21:19 2h43m bisect fix linux-6.1.y OK (0) job log log

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-1): P56/1:b..l
	(detected by 1, t=10503 jiffies, g=8561, q=256 ncpus=2)
task:kworker/u4:4    state:R  running task     stack:24400 pid:56    ppid:2      flags:0x00004000
Workqueue: writeback wb_workfn (flush-8:0)
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5245 [inline]
 __schedule+0x11d1/0x40e0 kernel/sched/core.c:6562
 preempt_schedule_irq+0xbb/0x160 kernel/sched/core.c:6874
 irqentry_exit+0x63/0x70 kernel/entry/common.c:439
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:691
RIP: 0010:lock_acquire+0x225/0x4a0 kernel/locking/lockdep.c:5666
Code: f7 84 24 80 00 00 00 00 02 00 00 43 c6 44 3d 04 f8 0f 85 f0 00 00 00 41 f7 c6 00 02 00 00 74 01 fb 48 c7 44 24 60 0e 36 e0 45 <4b> c7 44 3d 00 00 00 00 00 43 c7 44 3d 08 00 00 00 00 65 48 8b 04
RSP: 0018:ffffc90001576b00 EFLAGS: 00000206
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 3e9c355887893700
RDX: 0000000000000000 RSI: ffffffff8a8c2c20 RDI: ffffffff8adf2060
RBP: ffffc90001576c08 R08: dffffc0000000000 R09: 1ffffffff215fe48
R10: dffffc0000000000 R11: fffffbfff215fe49 R12: 0000000000000001
R13: 1ffff920002aed6c R14: 0000000000000246 R15: dffffc0000000000
 rcu_lock_acquire include/linux/rcupdate.h:350 [inline]
 rcu_read_lock include/linux/rcupdate.h:791 [inline]
 folio_memcg_lock+0x9e/0x320 mm/memcontrol.c:2101
 __folio_start_writeback+0x165/0x1000 mm/page-writeback.c:2985
 ext4_bio_write_page+0x30d/0x2ae0 fs/ext4/page-io.c:453
 mpage_submit_page+0x17a/0x210 fs/ext4/inode.c:2144
 mpage_map_and_submit_buffers fs/ext4/inode.c:2389 [inline]
 mpage_map_and_submit_extent fs/ext4/inode.c:2528 [inline]
 ext4_writepages+0x1b3d/0x2f40 fs/ext4/inode.c:2857
 do_writepages+0x3ba/0x640 mm/page-writeback.c:2491
 __writeback_single_inode+0x156/0x1160 fs/fs-writeback.c:1622
 writeback_sb_inodes+0xb30/0x1850 fs/fs-writeback.c:1913
 __writeback_inodes_wb+0x12a/0x3f0 fs/fs-writeback.c:1984
 wb_writeback+0x494/0xd50 fs/fs-writeback.c:2089
 wb_check_old_data_flush fs/fs-writeback.c:2189 [inline]
 wb_do_writeback fs/fs-writeback.c:2242 [inline]
 wb_workfn+0xb68/0xee0 fs/fs-writeback.c:2270
 process_one_work+0x8a2/0x1160 kernel/workqueue.c:2292
 worker_thread+0xaa2/0x1270 kernel/workqueue.c:2439
 kthread+0x29d/0x330 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
rcu: rcu_preempt kthread starved for 10546 jiffies! g8561 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:26416 pid:16    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5245 [inline]
 __schedule+0x11d1/0x40e0 kernel/sched/core.c:6562
 schedule+0xb9/0x180 kernel/sched/core.c:6638
 schedule_timeout+0x184/0x2d0 kernel/time/timer.c:2168
 rcu_gp_fqs_loop+0x303/0x1340 kernel/rcu/tree.c:1706
 rcu_gp_kthread+0x99/0x3b0 kernel/rcu/tree.c:1905
 kthread+0x29d/0x330 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 4322 Comm: syz-executor177 Not tainted syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/09/2026
RIP: 0010:arch_static_branch arch/x86/include/asm/jump_label.h:27 [inline]
RIP: 0010:static_key_false include/linux/jump_label.h:207 [inline]
RIP: 0010:native_write_msr arch/x86/include/asm/msr.h:147 [inline]
RIP: 0010:wrmsr arch/x86/include/asm/msr.h:254 [inline]
RIP: 0010:native_apic_msr_write+0x35/0x50 arch/x86/include/asm/apic.h:206
Code: 74 2a 83 ff 30 74 25 eb 10 81 ff d0 00 00 00 74 1b 81 ff e0 00 00 00 74 13 c1 ef 04 81 c7 00 08 00 00 89 f9 89 f0 31 d2 0f 30 <66> 90 c3 89 f6 31 d2 e9 5f c7 01 03 66 2e 0f 1f 84 00 00 00 00 00
RSP: 0018:ffffc90000007e38 EFLAGS: 00000046
RAX: 0000000000041f66 RBX: 0000000000041f66 RCX: 0000000000000838
RDX: 0000000000000000 RSI: 0000000000041f66 RDI: 0000000000000838
RBP: 0000000000000000 R08: ffffffff8e2029ef R09: 1ffffffff1c4053d
R10: dffffc0000000000 R11: fffffbfff1c4053e R12: dffffc0000000000
R13: 00000000100053e2 R14: ffffffff8c658b40 R15: dffffc0000000000
FS:  0000555564664480(0000) GS:ffff8880b8e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000200000000058 CR3: 0000000023cf3000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 apic_write arch/x86/include/asm/apic.h:394 [inline]
 lapic_next_event+0x5b/0x70 arch/x86/kernel/apic/apic.c:478
 clockevents_program_event+0x1bc/0x310 kernel/time/clockevents.c:334
 hrtimer_interrupt+0x59d/0x9c0 kernel/time/hrtimer.c:1890
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1107 [inline]
 __sysvec_apic_timer_interrupt+0x153/0x5a0 arch/x86/kernel/apic/apic.c:1124
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1118 [inline]
 sysvec_apic_timer_interrupt+0x9b/0xc0 arch/x86/kernel/apic/apic.c:1118
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:691
RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:160 [inline]
RIP: 0010:_raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:202
Code: 00 00 00 66 90 53 48 89 fb 48 83 c7 18 48 8b 74 24 08 e8 8e 52 2b f7 48 89 df e8 d6 1a 2c f7 e8 21 75 4f f7 fb bf 01 00 00 00 <e8> 26 7f 1f f7 65 8b 05 f7 fe c8 75 85 c0 74 02 5b c3 e8 44 13 c7
RSP: 0018:ffffc900043f7b98 EFLAGS: 00000282
RAX: 615e4ec8caeb5300 RBX: ffff88802e16ca00 RCX: 615e4ec8caeb5300
RDX: dffffc0000000000 RSI: ffffffff8a8c1aa0 RDI: 0000000000000001
RBP: ffff88802e16ce98 R08: ffff88802e16ca03 R09: 1ffff11005c2d940
R10: dffffc0000000000 R11: ffffed1005c2d941 R12: 1ffff11005c2d9d3
R13: 0000000000000021 R14: dffffc0000000000 R15: 0000000000000000
 spin_unlock_irq include/linux/spinlock.h:401 [inline]
 get_signal+0x1163/0x1350 kernel/signal.c:2874
 arch_do_signal_or_restart+0xc4/0x1350 arch/x86/kernel/signal.c:871
 exit_to_user_mode_loop+0x70/0x110 kernel/entry/common.c:174
 exit_to_user_mode_prepare+0xee/0x180 kernel/entry/common.c:210
 __syscall_exit_to_user_mode_work kernel/entry/common.c:292 [inline]
 syscall_exit_to_user_mode+0x16/0x40 kernel/entry/common.c:303
 do_syscall_64+0x58/0xa0 arch/x86/entry/common.c:82
 entry_SYSCALL_64_after_hwframe+0x68/0xd2
RIP: 0033:0x7f526df4a06e
Code: 08 0f 85 15 ec ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> 90 41 57 41 56 4d 89 c6 41 55 4d 89 cd 41 54 55 53 48 83 ec 08
RSP: 002b:00007fff8d3eac58 EFLAGS: 00000246
RAX: fffffffffffffff7 RBX: 0000555564664480 RCX: 00007f526df4a06e
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000d
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff8d3eacdc
R13: 00007fff8d3ead10 R14: 00007fff8d3eacf0 R15: 0000000000000001
 </TASK>
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.108 msecs
sched: RT throttling activated

Crashes (7):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2026/06/14 15:09 linux-6.1.y 228da13e907e 1d2f3589 .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: rcu detected stall in wb_workfn
2025/05/30 16:26 linux-6.1.y da3c5173c55f 3d2f584d .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: rcu detected stall in wb_workfn
2025/06/04 07:07 linux-6.1.y da3c5173c55f a30356b7 .config console log report syz / log [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: rcu detected stall in wb_workfn
2024/11/09 15:06 linux-6.1.y d7039b844a1c 6b856513 .config console log report syz / log [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: rcu detected stall in wb_workfn
2024/10/24 21:17 linux-6.1.y 7ec6f9fa3d97 0d144d1a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 BUG: soft lockup in wb_workfn
2026/02/20 00:53 linux-6.1.y 779f9571ac3e c8d8c52d .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: rcu detected stall in wb_workfn
2025/03/22 00:03 linux-6.1.y 344a09659766 c6512ef7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: rcu detected stall in wb_workfn
* Struck through repros no longer work on HEAD.