syzbot


INFO: rcu detected stall in schedule_timeout (7)

Status: upstream: reported syz repro on 2024/11/09 08:24
Subsystems: kernel
[Documentation on labels]
Reported-by: syzbot+8926d1b522e7194a4b3e@syzkaller.appspotmail.com
First crash: 156d, last: 22d
Cause bisection: failed (error log, bisect log)
  
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [kernel?] INFO: rcu detected stall in schedule_timeout (7) 0 (1) 2024/11/09 08:24
Similar bugs (8)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in schedule_timeout (3) cgroups mm 11 1905d 1906d 0/28 closed as invalid on 2020/01/09 08:13
upstream INFO: rcu detected stall in schedule_timeout (2) kernel 2 1906d 1906d 0/28 closed as invalid on 2020/01/08 05:23
upstream INFO: rcu detected stall in schedule_timeout (4) kernel 1 611d 611d 0/28 closed as invalid on 2023/09/01 06:44
linux-6.1 INFO: rcu detected stall in schedule_timeout 28 279d 344d 0/3 auto-obsoleted due to no activity on 2024/08/31 04:44
upstream INFO: rcu detected stall in schedule_timeout (5) kernel 2 516d 551d 0/28 auto-obsoleted due to no activity on 2024/01/26 16:31
upstream INFO: rcu detected stall in schedule_timeout cgroups mm 69 1940d 1942d 0/28 closed as invalid on 2019/12/04 14:14
upstream INFO: rcu detected stall in schedule_timeout (6) usb C done 40 158d 315d 28/28 fixed on 2024/10/22 11:57
linux-5.15 INFO: rcu detected stall in schedule_timeout origin:upstream C 8 7d19h 309d 0/3 upstream: reported C repro on 2024/05/22 14:45
Last patch testing requests (4)
Created Duration User Patch Repo Result
2025/02/03 20:53 24m retest repro upstream report log
2025/02/03 20:53 24m retest repro upstream OK log
2025/02/03 20:53 31m retest repro upstream report log
2024/11/19 08:30 30m retest repro upstream report log

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-1): P5946/1:b..l
rcu: 	(detected by 0, t=10502 jiffies, g=10141, q=524 ncpus=2)
task:udevd           state:R  running task     stack:26016 pid:5946  tgid:5946  ppid:5199   flags:0x00004002
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5369 [inline]
 __schedule+0x1850/0x4c30 kernel/sched/core.c:6756
 preempt_schedule_common+0x84/0xd0 kernel/sched/core.c:6935
 preempt_schedule+0xe1/0xf0 kernel/sched/core.c:6959
 preempt_schedule_thunk+0x1a/0x30 arch/x86/entry/thunk.S:12
 __raw_spin_unlock include/linux/spinlock_api_smp.h:143 [inline]
 _raw_spin_unlock+0x3e/0x50 kernel/locking/spinlock.c:186
 spin_unlock include/linux/spinlock.h:391 [inline]
 zap_pte_range mm/memory.c:1705 [inline]
 zap_pmd_range mm/memory.c:1753 [inline]
 zap_pud_range mm/memory.c:1782 [inline]
 zap_p4d_range mm/memory.c:1803 [inline]
 unmap_page_range+0x36c3/0x4040 mm/memory.c:1824
 unmap_vmas+0x3cc/0x5f0 mm/memory.c:1914
 exit_mmap+0x275/0xc20 mm/mmap.c:1668
 __mmput+0x115/0x3c0 kernel/fork.c:1348
 exit_mm+0x220/0x310 kernel/exit.c:570
 do_exit+0x9ad/0x28e0 kernel/exit.c:925
 do_group_exit+0x207/0x2c0 kernel/exit.c:1087
 __do_sys_exit_group kernel/exit.c:1098 [inline]
 __se_sys_exit_group kernel/exit.c:1096 [inline]
 __x64_sys_exit_group+0x3f/0x40 kernel/exit.c:1096
 x64_sys_call+0x26a8/0x26b0 arch/x86/include/generated/asm/syscalls_64.h:232
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7ff85fefca90
RSP: 002b:00007ffc56974c18 EFLAGS: 00000206 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff85fefca90
RDX: 00000000000000e7 RSI: 000000000000003c RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000007 R09: 1d18297dccdc26d6
R10: 00000000ffffffff R11: 0000000000000206 R12: 000055b1a750ec40
R13: 00007ffc56974c58 R14: 0000000000000001 R15: 000055b1a74e9910
 </TASK>
rcu: rcu_preempt kthread starved for 7711 jiffies! g10141 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:25976 pid:17    tgid:17    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5369 [inline]
 __schedule+0x1850/0x4c30 kernel/sched/core.c:6756
 preempt_schedule_irq+0xfb/0x1c0 kernel/sched/core.c:7078
 irqentry_exit+0x5e/0x90 kernel/entry/common.c:354
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:blk_flush_plug include/linux/blkdev.h:1060 [inline]
RIP: 0010:sched_submit_work kernel/sched/core.c:6812 [inline]
RIP: 0010:schedule+0x109/0x320 kernel/sched/core.c:6847
Code: eb 08 48 89 ef e8 47 b3 9d f5 4c 8d bd 78 12 00 00 4c 89 f8 48 c1 e8 03 42 80 3c 20 00 74 08 4c 89 ff e8 6a 26 3c f6 49 8b 3f <48> 85 ff 74 0a be 01 00 00 00 e8 58 f3 d7 f8 48 c7 c7 a0 53 7f 8e
RSP: 0018:ffffc90000167b68 EFLAGS: 00000246
RAX: 1ffff1100395324f RBX: ffff88801ca98000 RCX: ffff88801ca98ad8
RDX: dffffc0000000000 RSI: ffffffff8c0aaae0 RDI: 0000000000000000
RBP: ffff88801ca98000 R08: ffffffff942a4897 R09: 1ffffffff2854912
R10: dffffc0000000000 R11: fffffbfff2854913 R12: dffffc0000000000
R13: 1ffff9200002cf74 R14: ffffffff8bc9b500 R15: ffff88801ca99278
 schedule_timeout+0x15a/0x290 kernel/time/sleep_timeout.c:99
 rcu_gp_fqs_loop+0x2df/0x1330 kernel/rcu/tree.c:2045
 rcu_gp_kthread+0xa7/0x3b0 kernel/rcu/tree.c:2247
 kthread+0x2f2/0x390 kernel/kthread.c:389
 ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
CPU: 0 UID: 0 PID: 6035 Comm: kworker/0:5 Not tainted 6.13.0-rc4-syzkaller-00012-g9b2ffa6148b1 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: wg-crypt-wg1 wg_packet_tx_worker
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0xd8/0x140 kernel/locking/spinlock.c:194
Code: 9c 8f 44 24 20 42 80 3c 23 00 74 08 4c 89 f7 e8 6e 80 3a f6 f6 44 24 21 02 75 52 41 f7 c7 00 02 00 00 74 01 fb bf 01 00 00 00 <e8> 13 27 a2 f5 65 8b 05 74 7a 38 74 85 c0 74 43 48 c7 04 24 0e 36
RSP: 0018:ffffc900000078c0 EFLAGS: 00000206
RAX: fb9b9240a3434400 RBX: 1ffff92000000f1c RCX: ffffffff817b279a
RDX: dffffc0000000000 RSI: ffffffff8c0a98e0 RDI: 0000000000000001
RBP: ffffc90000007950 R08: ffffffff942a498f R09: 1ffffffff2854931
R10: dffffc0000000000 R11: fffffbfff2854932 R12: dffffc0000000000
R13: 1ffff92000000f18 R14: ffffc900000078e0 R15: 0000000000000246
FS:  0000000000000000(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055b1a7503d88 CR3: 0000000033d72000 CR4: 0000000000350ef0
Call Trace:
 <IRQ>
 dummy_timer+0x44d6/0x4620
 __run_hrtimer kernel/time/hrtimer.c:1739 [inline]
 __hrtimer_run_queues+0x59d/0xd30 kernel/time/hrtimer.c:1803
 hrtimer_run_softirq+0x19a/0x2c0 kernel/time/hrtimer.c:1820
 handle_softirqs+0x2d6/0x9b0 kernel/softirq.c:561
 __do_softirq kernel/softirq.c:595 [inline]
 invoke_softirq kernel/softirq.c:435 [inline]
 __irq_exit_rcu+0xf7/0x220 kernel/softirq.c:662
 irq_exit_rcu+0x9/0x30 kernel/softirq.c:678
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1049 [inline]
 sysvec_apic_timer_interrupt+0xa6/0xc0 arch/x86/kernel/apic/apic.c:1049
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:srso_alias_safe_ret+0x0/0x7 arch/x86/lib/retpoline.S:171
Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc <48> 8d 64 24 08 c3 cc e8 f4 ff ff ff 0f 0b cc cc cc cc cc cc cc cc
RSP: 0018:ffffc90003fffb08 EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff888011cee4d8 RCX: ffffffff8676a782
RDX: ffff8880302d3c00 RSI: 0000000000000003 RDI: 0000000000000001
RBP: 0000000000000003 R08: ffffffff8676a79c R09: 1ffff1100239dc9b
R10: dffffc0000000000 R11: ffffed100239dc9c R12: dffffc0000000000
R13: 1000000000000000 R14: ffff888033929318 R15: ffff888011cee000
 srso_alias_return_thunk+0x5/0xfbef5 arch/x86/lib/retpoline.S:181
 raw_atomic_fetch_sub_release include/linux/atomic/atomic-arch-fallback.h:949 [inline]
 atomic_fetch_sub_release include/linux/atomic/atomic-instrumented.h:401 [inline]
 __refcount_sub_and_test include/linux/refcount.h:264 [inline]
 __refcount_dec_and_test include/linux/refcount.h:307 [inline]
 refcount_dec_and_test include/linux/refcount.h:325 [inline]
 kref_put include/linux/kref.h:64 [inline]
 wg_noise_keypair_put+0x5c/0x290 drivers/net/wireguard/noise.c:139
 wg_packet_tx_worker+0x5fe/0x810 drivers/net/wireguard/send.c:280
 process_one_work kernel/workqueue.c:3229 [inline]
 process_scheduled_works+0xa68/0x1840 kernel/workqueue.c:3310
 worker_thread+0x870/0xd30 kernel/workqueue.c:3391
 kthread+0x2f2/0x390 kernel/kthread.c:389
 ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Crashes (15):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/12/25 00:39 upstream 9b2ffa6148b1 444551c4 .config console log report syz / log [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in schedule_timeout
2024/11/05 08:01 upstream 557329bcecc2 509da429 .config console log report syz / log [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in schedule_timeout
2025/03/05 16:23 upstream 48a5eed9ad58 60f5d8d9 .config console log report info [disk image] [vmlinux] [kernel image] ci-qemu-gce-upstream-auto INFO: rcu detected stall in schedule_timeout
2025/02/28 05:07 upstream 1e15510b71c9 6a8fcbc4 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in schedule_timeout
2025/02/21 12:54 upstream 27eddbf34490 0808a665 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in schedule_timeout
2025/02/09 12:23 upstream 9946eaf552b1 ef44b750 .config console log report info [disk image] [vmlinux] [kernel image] ci-qemu-gce-upstream-auto INFO: rcu detected stall in schedule_timeout
2025/02/06 06:41 upstream 92514ef226f5 577d049b .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in schedule_timeout
2025/01/20 19:45 upstream ffd294d346d1 6e87cfa2 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in schedule_timeout
2025/01/17 21:40 upstream 9bffa1ad25b8 953d1c45 .config console log report syz / log [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in schedule_timeout
2025/01/17 14:57 upstream 9bffa1ad25b8 953d1c45 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in schedule_timeout
2025/01/07 05:57 upstream fbfd64d25c7a f3558dbf .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in schedule_timeout
2025/01/02 14:55 upstream 56e6a3499e14 d3ccff63 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in schedule_timeout
2024/12/16 14:46 upstream 78d4f34e2115 f93b2b55 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in schedule_timeout
2024/10/22 14:24 upstream c2ee9f594da8 9d74f456 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in schedule_timeout
2024/10/26 16:52 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing c6d9e43954bf 65e8686b .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in schedule_timeout
* Struck through repros no longer work on HEAD.