syzbot


INFO: rcu detected stall in ext4_file_write_iter (6)

Status: fixed on 2023/06/08 14:41
Subsystems: ext4
[Documentation on labels]
Reported-by: syzbot+b9564ba6e8e00694511b@syzkaller.appspotmail.com
Fix commit: d125d1349abe alarmtimer: Prevent starvation by small intervals and SIG_IGN
First crash: 590d, last: 379d
Cause bisection: failed (error log, bisect log)
  
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] INFO: rcu detected stall in ext4_file_write_iter (6) 3 (4) 2023/02/14 10:27
Similar bugs (14)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in ext4_file_write_iter (8) mm 3 132d 188d 0/26 auto-obsoleted due to no activity on 2024/03/14 12:59
linux-5.15 INFO: rcu detected stall in ext4_file_write_iter 1 156d 156d 0/3 auto-obsoleted due to no activity on 2024/02/29 20:44
upstream INFO: rcu detected stall in ext4_file_write_iter (3) block 5 990d 1080d 0/26 auto-closed as invalid on 2021/11/07 19:52
upstream INFO: rcu detected stall in ext4_file_write_iter (5) mm 3 730d 731d 0/26 auto-closed as invalid on 2022/06/25 07:58
linux-6.1 INFO: rcu detected stall in ext4_file_write_iter 1 25d 25d 0/3 upstream: reported on 2024/04/01 06:27
upstream INFO: rcu detected stall in ext4_file_write_iter (7) mm bpf ext4 3 238d 280d 23/26 fixed on 2023/10/12 12:47
upstream INFO: rcu detected stall in ext4_file_write_iter (4) mm C unreliable 58 826d 855d 0/26 closed as invalid on 2022/02/08 10:32
android-49 INFO: rcu detected stall in ext4_file_write_iter syz 2 1788d 1829d 0/3 public: reported syz repro on 2019/04/23 08:58
upstream INFO: rcu detected stall in ext4_file_write_iter (2) ext4 1 1291d 1291d 0/26 auto-closed as invalid on 2021/01/10 12:58
linux-4.14 INFO: rcu detected stall in ext4_file_write_iter C 7 431d 1840d 0/1 upstream: reported C repro on 2019/04/12 16:30
upstream INFO: rcu detected stall in ext4_file_write_iter C inconclusive done 93 1353d 1886d 15/26 fixed on 2020/09/25 01:17
linux-4.19 INFO: rcu detected stall in ext4_file_write_iter C error 9 491d 1832d 0/1 upstream: reported C repro on 2019/04/20 13:09
linux-4.19 BUG: soft lockup in ext4_file_write_iter 1 1036d 1036d 0/1 auto-closed as invalid on 2021/10/22 22:26
android-5-15 BUG: soft lockup in ext4_file_write_iter 3 7d18h 19d 0/2 premoderation: reported on 2024/04/06 14:01
Last patch testing requests (6)
Created Duration User Patch Repo Result
2023/02/10 06:51 21m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 4fafd96910ad OK log
2023/02/09 07:31 15m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 4fafd96910ad report log
2023/02/09 07:27 5m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 4fafd96910ad error OK
2023/02/09 06:19 21m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 4fafd96910ad error OK
2023/02/09 01:07 15m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 4fafd96910ad report log
2023/02/08 13:26 15m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 4fafd96910ad report log

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	(detected by 1, t=10502 jiffies, g=5605, q=143 ncpus=2)
rcu: All QSes seen, last rcu_preempt kthread activity 10502 (4294955582-4294945080), jiffies_till_next_fqs=1, root ->qsmask 0x0
rcu: rcu_preempt kthread starved for 10502 jiffies! g5605 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:28680 pid:16    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5301 [inline]
 __schedule+0x245a/0x5a60 kernel/sched/core.c:6619
 schedule+0xde/0x1a0 kernel/sched/core.c:6695
 schedule_timeout+0x14e/0x2b0 kernel/time/timer.c:2167
 rcu_gp_fqs_loop+0x190/0x910 kernel/rcu/tree.c:1609
 rcu_gp_kthread+0x23a/0x360 kernel/rcu/tree.c:1808
 kthread+0x2e8/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 5108 Comm: syz-executor192 Not tainted 6.2.0-rc6-next-20230203-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/12/2023
RIP: 0010:__sanitizer_cov_trace_const_cmp1+0x0/0x20 kernel/kcov.c:289
Code: 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 8b 0c 24 48 89 f2 48 89 fe bf 06 00 00 00 e9 f8 fe ff ff 0f 1f 84 00 00 00 00 00 <f3> 0f 1e fa 48 8b 0c 24 40 0f b6 d6 40 0f b6 f7 bf 01 00 00 00 e9
RSP: 0018:ffffc90000007dd8 EFLAGS: 00000046
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffffffff81757653
RDX: fffffbfff1cea2eb RSI: 0000000000000001 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff8e751757
R10: fffffbfff1cea2ea R11: 0000000000000000 R12: 0000002a9fc7f679
R13: 0000000000000001 R14: ffffffff91cb6e28 R15: ffff8880775e6f60
FS:  0000555557516300(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8f88861130 CR3: 0000000072b77000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 trace_alarmtimer_fired include/trace/events/alarmtimer.h:73 [inline]
 alarmtimer_fired+0x327/0x670 kernel/time/alarmtimer.c:220
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x600/0xcf0 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x320/0x7b0 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1096 [inline]
 __sysvec_apic_timer_interrupt+0x180/0x660 arch/x86/kernel/apic/apic.c:1113
 sysvec_apic_timer_interrupt+0x92/0xc0 arch/x86/kernel/apic/apic.c:1107
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:645
RIP: 0010:lock_is_held_type+0x103/0x140 kernel/locking/lockdep.c:5767
Code: 00 00 b8 ff ff ff ff 65 0f c1 05 48 c8 ef 75 83 f8 01 75 29 9c 58 f6 c4 02 75 3d 48 f7 04 24 00 02 00 00 74 01 fb 48 83 c4 08 <44> 89 e8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 45 31 ed eb b9 0f 0b 48
RSP: 0018:ffffc90003d0f3c8 EFLAGS: 00000282
RAX: 0000000000000046 RBX: 0000000000000002 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffffff8c794680 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffff8880773d8000
R13: 0000000000000000 R14: 00000000ffffffff R15: ffff8880773d8a60
 lock_is_held include/linux/lockdep.h:288 [inline]
 rcu_read_lock_sched_held+0x3e/0x70 kernel/rcu/update.c:125
 trace_block_dirty_buffer include/trace/events/block.h:58 [inline]
 mark_buffer_dirty+0x445/0x570 fs/buffer.c:1126
 __block_commit_write.constprop.0.isra.0+0x19d/0x330 fs/buffer.c:2108
 block_write_end+0x53/0x220 fs/buffer.c:2187
 generic_write_end+0x8d/0x440 fs/buffer.c:2201
 ext4_da_write_end+0x1f5/0xa50 fs/ext4/inode.c:3174
 generic_perform_write+0x316/0x570 mm/filemap.c:3781
 ext4_buffered_write_iter+0x15b/0x460 fs/ext4/file.c:285
 ext4_file_write_iter+0x8bf/0x1710 fs/ext4/file.c:700
 call_write_iter include/linux/fs.h:1851 [inline]
 do_iter_readv_writev+0x20b/0x3b0 fs/read_write.c:735
 do_iter_write+0x182/0x700 fs/read_write.c:861
 vfs_iter_write+0x74/0xa0 fs/read_write.c:902
 iter_file_splice_write+0x745/0xc90 fs/splice.c:686
 do_splice_from fs/splice.c:764 [inline]
 direct_splice_actor+0x114/0x180 fs/splice.c:931
 splice_direct_to_actor+0x335/0x8a0 fs/splice.c:886
 do_splice_direct+0x1ab/0x280 fs/splice.c:974
 do_sendfile+0xb19/0x12c0 fs/read_write.c:1255
 __do_sys_sendfile64 fs/read_write.c:1317 [inline]
 __se_sys_sendfile64 fs/read_write.c:1309 [inline]
 __x64_sys_sendfile64+0x14d/0x210 fs/read_write.c:1309
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f8f887f06a9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 14 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 c0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffccec967c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028
RAX: ffffffffffffffda RBX: 0000000000000062 RCX: 00007f8f887f06a9
RDX: 00000000200001c0 RSI: 0000000000000003 RDI: 0000000000000003
RBP: 0000000000000000 R08: 00007ffccec967f0 R09: 00007ffccec967f0
R10: 000000000200a198 R11: 0000000000000246 R12: 00007ffccec967ec
R13: 00007ffccec96820 R14: 00007ffccec96800 R15: 0000000000000003
 </TASK>
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.925 msecs

Crashes (8):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/02/04 10:21 linux-next 4fafd96910ad 1b2f701a .config strace log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in ext4_file_write_iter
2023/04/11 00:27 upstream 0d3eb744aed4 71147e29 .config console log report info ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in ext4_file_write_iter
2022/12/21 14:36 upstream b6bb9676f216 4067838e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in ext4_file_write_iter
2022/11/24 09:48 upstream 4312098baf37 12c66417 .config console log report info ci-upstream-kasan-gce-root INFO: rcu detected stall in ext4_file_write_iter
2022/10/14 06:02 upstream 55be6084c8e0 4954e4b2 .config console log report info [disk image] [vmlinux] ci-upstream-kasan-gce-root INFO: rcu detected stall in ext4_file_write_iter
2022/11/01 07:03 linux-next 4d48f589d294 a1d8560a .config console log report info [disk image] [vmlinux] ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in ext4_file_write_iter
2022/09/13 11:20 linux-next e47eb90a0a9a a08652b0 .config console log report info ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in ext4_file_write_iter
2023/04/12 08:11 upstream e62252bc55b6 1a1596b6 .config console log report info ci-qemu2-arm64-compat BUG: soft lockup in ext4_file_write_iter
* Struck through repros no longer work on HEAD.