syzbot


INFO: rcu detected stall in fq_pie_timer (2)

Status: upstream: reported C repro on 2023/12/27 13:54
Subsystems: net
[Documentation on labels]
Reported-by: syzbot+15c3ba3f7ca8ced0914d@syzkaller.appspotmail.com
First crash: 355d, last: 19d
Cause bisection: introduced by (bisect log) :
commit ec97ecf1ebe485a17cd8395a5f35e6b80b57665a
Author: Mohit P. Tahiliani <tahiliani@nitk.edu.in>
Date: Wed Jan 22 18:22:33 2020 +0000

  net: sched: add Flow Queue PIE packet scheduler

Crash: INFO: rcu detected stall in fq_pie_timer (log)
Repro: C syz .config
  
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [net?] INFO: rcu detected stall in fq_pie_timer (2) 2 (5) 2023/12/29 11:10
Similar bugs (7)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in fq_pie_timer net C error 35 1050d 1268d 20/28 fixed on 2022/03/08 16:11
linux-5.15 INFO: rcu detected stall in fq_pie_timer C done 23 479d 569d 0/3 auto-obsoleted due to no activity on 2024/10/31 18:20
upstream BUG: soft lockup in fq_pie_timer (2) net C error done 3 596d 662d 0/28 closed as invalid on 2023/05/26 05:46
upstream BUG: soft lockup in fq_pie_timer net C error error 10 792d 829d 0/28 closed as invalid on 2022/11/18 11:06
linux-6.1 BUG: soft lockup in fq_pie_timer (2) origin:upstream C inconclusive 5 236d 471d 0/3 upstream: reported C repro on 2023/08/29 01:49
upstream BUG: soft lockup in fq_pie_timer (4) net syz 5 457d 507d 23/28 fixed on 2023/10/12 12:48
linux-6.1 BUG: soft lockup in fq_pie_timer C done 7 551d 559d 3/3 fixed on 2023/07/22 07:09
Last patch testing requests (6)
Created Duration User Patch Repo Result
2024/10/05 05:14 44m retest repro upstream OK log
2024/06/28 06:17 26m retest repro net OK log
2024/04/18 17:47 8h47m retest repro net error
2024/01/06 15:32 19m retest repro net report log
2023/12/29 10:46 23m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master report log
2023/12/28 12:07 19m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master report log
Fix bisection attempts (3)
Created Duration User Patch Repo Result
2024/11/21 14:52 6h25m bisect fix net OK (0) job log log
2024/10/21 05:00 3h20m bisect fix net OK (0) job log log
2024/02/07 18:25 4h05m bisect fix net OK (0) job log log

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	0-...!: (1 ticks this GP) idle=e4c4/1/0x4000000000000000 softirq=78631/78631 fqs=3
rcu: 	(detected by 1, t=10502 jiffies, g=73289, q=469 ncpus=2)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 11831 Comm: kworker/u8:11 Not tainted 6.12.0-syzkaller-07834-g06afb0f36106 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: bat_events batadv_nc_worker
RIP: 0010:variable_test_bit arch/x86/include/asm/bitops.h:227 [inline]
RIP: 0010:arch_test_bit arch/x86/include/asm/bitops.h:239 [inline]
RIP: 0010:_test_bit include/asm-generic/bitops/instrumented-non-atomic.h:142 [inline]
RIP: 0010:cpumask_test_cpu include/linux/cpumask.h:570 [inline]
RIP: 0010:cpu_online include/linux/cpumask.h:1117 [inline]
RIP: 0010:trace_hrtimer_expire_exit include/trace/events/timer.h:306 [inline]
RIP: 0010:__run_hrtimer kernel/time/hrtimer.c:1742 [inline]
RIP: 0010:__hrtimer_run_queues+0x293/0xae0 kernel/time/hrtimer.c:1803
Code: 00 83 fd 07 0f 87 06 07 00 00 e8 88 c8 12 00 89 ed be 08 00 00 00 48 89 e8 48 c1 e8 06 48 8d 3c c5 d0 3e 5f 90 e8 9d 92 75 00 <48> 0f a3 2d 95 cd e3 0e 40 0f 92 c5 31 ff 89 ee e8 38 ca 12 00 40
RSP: 0018:ffffc900000078e8 EFLAGS: 00000046
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffffffff817b7133
RDX: fffffbfff20be7db RSI: 0000000000000008 RDI: ffffffff905f3ed0
RBP: 0000000000000000 R08: 0000000000000000 R09: fffffbfff20be7da
R10: ffffffff905f3ed7 R11: 0000000000000006 R12: ffff88805c2c8340
R13: ffff8880b862cb00 R14: ffff8880b862ca00 R15: ffffffff893130f0
FS:  0000000000000000(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000110c3a98f7 CR3: 00000000122c6000 CR4: 00000000003526f0
DR0: 0000000000000003 DR1: 0000000000000004 DR2: 0000000100000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 hrtimer_interrupt+0x392/0x8e0 kernel/time/hrtimer.c:1865
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1038 [inline]
 __sysvec_apic_timer_interrupt+0x10f/0x400 arch/x86/kernel/apic/apic.c:1055
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1049 [inline]
 sysvec_apic_timer_interrupt+0x52/0xc0 arch/x86/kernel/apic/apic.c:1049
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:pie_vars_init include/net/pie.h:103 [inline]
RIP: 0010:pie_calculate_probability+0x6bc/0x850 net/sched/sch_pie.c:414
Code: ff ff e8 d7 d5 5e f8 48 8d 7b 10 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 86 01 00 00 48 8d 7b 18 <48> c7 43 10 46 c3 23 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48
RSP: 0018:ffffc90000007b78 EFLAGS: 00000246
RAX: dffffc0000000000 RBX: ffffc9001c726c30 RCX: ffffffff892f637a
RDX: 1ffff920038e4d88 RSI: ffffffff892f63c9 RDI: ffffc9001c726c48
RBP: ffff88805baf9300 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000005 R12: ffffc9001c726c68
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 fq_pie_timer+0x215/0x5a0 net/sched/sch_fq_pie.c:398
 call_timer_fn+0x1a0/0x610 kernel/time/timer.c:1793
 expire_timers kernel/time/timer.c:1844 [inline]
 __run_timers+0x6e8/0x930 kernel/time/timer.c:2418
 __run_timer_base kernel/time/timer.c:2430 [inline]
 __run_timer_base kernel/time/timer.c:2422 [inline]
 run_timer_base+0x114/0x190 kernel/time/timer.c:2439
 run_timer_softirq+0x1a/0x40 kernel/time/timer.c:2449
 handle_softirqs+0x213/0x8f0 kernel/softirq.c:554
 __do_softirq kernel/softirq.c:588 [inline]
 invoke_softirq kernel/softirq.c:428 [inline]
 __irq_exit_rcu+0x109/0x170 kernel/softirq.c:655
 irq_exit_rcu+0x9/0x30 kernel/softirq.c:671
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1049 [inline]
 sysvec_apic_timer_interrupt+0xa4/0xc0 arch/x86/kernel/apic/apic.c:1049
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:lock_acquire.part.0+0x155/0x380 kernel/locking/lockdep.c:5814
Code: b8 ff ff ff ff 65 0f c1 05 c0 8d 97 7e 83 f8 01 0f 85 d0 01 00 00 9c 58 f6 c4 02 0f 85 e5 01 00 00 48 85 ed 0f 85 b6 01 00 00 <48> b8 00 00 00 00 00 fc ff df 48 01 c3 48 c7 03 00 00 00 00 48 c7
RSP: 0018:ffffc9000344faa8 EFLAGS: 00000206
RAX: 0000000000000046 RBX: 1ffff92000689f56 RCX: 00000000d13a4ef6
RDX: 0000000000000001 RSI: ffffffff8b6cdda0 RDI: ffffffff8bd0db80
RBP: 0000000000000200 R08: 0000000000000000 R09: fffffbfff2dc8398
R10: ffffffff96e41cc7 R11: 0000000000000002 R12: 0000000000000000
R13: ffffffff8e1bbd80 R14: 0000000000000000 R15: 0000000000000000
 rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
 rcu_read_lock include/linux/rcupdate.h:849 [inline]
 batadv_nc_purge_orig_hash net/batman-adv/network-coding.c:408 [inline]
 batadv_nc_worker+0x16a/0x1060 net/batman-adv/network-coding.c:719
 process_one_work+0x9c5/0x1ba0 kernel/workqueue.c:3229
 process_scheduled_works kernel/workqueue.c:3310 [inline]
 worker_thread+0x6c8/0xf00 kernel/workqueue.c:3391
 kthread+0x2c1/0x3a0 kernel/kthread.c:389
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
rcu: rcu_preempt kthread starved for 10496 jiffies! g73289 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:27616 pid:17    tgid:17    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5369 [inline]
 __schedule+0xe58/0x5ad0 kernel/sched/core.c:6756
 __schedule_loop kernel/sched/core.c:6833 [inline]
 schedule+0xe7/0x350 kernel/sched/core.c:6848
 schedule_timeout+0x124/0x280 kernel/time/sleep_timeout.c:99
 rcu_gp_fqs_loop+0x1eb/0xb00 kernel/rcu/tree.c:2045
 rcu_gp_kthread+0x271/0x380 kernel/rcu/tree.c:2247
 kthread+0x2c1/0x3a0 kernel/kthread.c:389
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
CPU: 1 UID: 0 PID: 11 Comm: kworker/u8:0 Not tainted 6.12.0-syzkaller-07834-g06afb0f36106 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:340 [inline]
RIP: 0010:smp_call_function_many_cond+0x458/0x1300 kernel/smp.c:884
Code: 0c 00 85 ed 74 4d 48 b8 00 00 00 00 00 fc ff df 4d 89 f4 4c 89 f5 49 c1 ec 03 83 e5 07 49 01 c4 83 c5 03 e8 ba 11 0c 00 f3 90 <41> 0f b6 04 24 40 38 c5 7c 08 84 c0 0f 85 a7 0c 00 00 8b 43 08 31
RSP: 0018:ffffc90000107998 EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff8880b86468a0 RCX: ffffffff8182280c
RDX: ffff88801c6a2440 RSI: ffffffff818227e6 RDI: 0000000000000005
RBP: 0000000000000003 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000006 R12: ffffed10170c8d15
R13: 0000000000000001 R14: ffff8880b86468a8 R15: ffff8880b873fd80
FS:  0000000000000000(0000) GS:ffff8880b8700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020010000 CR3: 000000000df7e000 CR4: 00000000003526f0
DR0: 0000000000000003 DR1: 0000000000000004 DR2: 0000000100000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 </IRQ>
 <TASK>
 on_each_cpu_cond_mask+0x40/0x90 kernel/smp.c:1051
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:2085 [inline]
 text_poke_bp_batch+0x22b/0x760 arch/x86/kernel/alternative.c:2295
 text_poke_flush arch/x86/kernel/alternative.c:2486 [inline]
 text_poke_flush arch/x86/kernel/alternative.c:2483 [inline]
 text_poke_finish+0x30/0x40 arch/x86/kernel/alternative.c:2493
 arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
 jump_label_update+0x1d7/0x400 kernel/jump_label.c:920
 static_key_enable_cpuslocked+0x1b7/0x270 kernel/jump_label.c:210
 static_key_enable+0x1a/0x20 kernel/jump_label.c:223
 toggle_allocation_gate mm/kfence/core.c:849 [inline]
 toggle_allocation_gate+0xfc/0x260 mm/kfence/core.c:841
 process_one_work+0x9c5/0x1ba0 kernel/workqueue.c:3229
 process_scheduled_works kernel/workqueue.c:3310 [inline]
 worker_thread+0x6c8/0xf00 kernel/workqueue.c:3391
 kthread+0x2c1/0x3a0 kernel/kthread.c:389
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Crashes (34):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/11/23 11:21 upstream 06afb0f36106 68da6d95 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/09/21 04:35 upstream a430d95c5efa 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/09/13 19:59 upstream e936e7d4a83b 158f4851 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in fq_pie_timer
2024/09/13 11:27 upstream 196145c606d0 73e8a465 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2024/09/10 20:13 upstream 8d8d276ba2fb 79eef296 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in fq_pie_timer
2024/09/08 05:34 upstream d1f2d51b711a 9750182a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/09/03 23:13 upstream 88fac17500f4 9d47f20a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/09/03 22:46 upstream 88fac17500f4 9d47f20a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in fq_pie_timer
2024/08/30 10:15 upstream 20371ba12063 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/30 10:10 upstream 20371ba12063 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/30 10:10 upstream 20371ba12063 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/30 10:09 upstream 20371ba12063 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/30 10:08 upstream 20371ba12063 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/30 08:28 upstream d5d547aa7b51 54fe8471 .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/30 05:21 upstream d5d547aa7b51 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/30 05:20 upstream d5d547aa7b51 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/29 20:28 upstream d5d547aa7b51 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/29 20:27 upstream d5d547aa7b51 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/29 20:27 upstream d5d547aa7b51 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/29 20:25 upstream d5d547aa7b51 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/29 20:13 upstream d5d547aa7b51 54fe8471 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/26 13:14 upstream 5be63fc19fca 9aee4e0b .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/08/25 15:09 upstream 5be63fc19fca d7d32352 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/07/25 11:32 upstream c33ffdb70cc6 b24754ac .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2024/07/17 12:17 upstream 51835949dda3 215bec2d .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2024/07/14 08:27 upstream d0d0cd380055 eaeb5c15 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2024/06/28 13:16 upstream 5bbd9b249880 b62c7d46 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in fq_pie_timer
2024/05/25 09:04 upstream 0b32d436c015 a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: rcu detected stall in fq_pie_timer
2023/12/23 13:47 net 7c5e046bdcb2 fb427a07 .config console log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: rcu detected stall in fq_pie_timer
2024/06/11 09:27 linux-next a957267fa7e9 048c640a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in fq_pie_timer
2024/05/13 12:24 linux-next 6ba6c795dc73 9026e142 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in fq_pie_timer
2024/08/19 20:15 upstream b0da640826ba 9f0ab3fb .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-badwrites-root BUG: soft lockup in fq_pie_timer
2024/05/28 18:09 upstream 2bfcfd584ff5 34889ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-badwrites-root BUG: soft lockup in fq_pie_timer
2024/05/25 06:51 upstream 0b32d436c015 a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-badwrites-root BUG: soft lockup in fq_pie_timer
* Struck through repros no longer work on HEAD.