syzbot


INFO: rcu detected stall in fq_pie_timer

Status: fixed on 2022/03/08 16:11
Subsystems: net
[Documentation on labels]
Fix commit: 61c2402665f1 net/sched: fq_pie: prevent dismantle issue
First crash: 1039d, last: 821d
Cause bisection: failed (error log, bisect log)
  
Similar bugs (7)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in fq_pie_timer (2) net C done 1 79d 121d 0/26 upstream: reported C repro on 2023/12/27 13:54
linux-5.15 INFO: rcu detected stall in fq_pie_timer origin:lts-only C done 23 250d 340d 0/3 upstream: reported C repro on 2023/05/22 23:26
upstream BUG: soft lockup in fq_pie_timer (2) net C error done 3 367d 433d 0/26 closed as invalid on 2023/05/26 05:46
upstream BUG: soft lockup in fq_pie_timer net C error error 10 563d 600d 0/26 closed as invalid on 2022/11/18 11:06
linux-6.1 BUG: soft lockup in fq_pie_timer (2) origin:upstream C inconclusive 5 7d20h 242d 0/3 upstream: reported C repro on 2023/08/29 01:49
upstream BUG: soft lockup in fq_pie_timer (4) net syz 5 228d 278d 23/26 fixed on 2023/10/12 12:48
linux-6.1 BUG: soft lockup in fq_pie_timer C done 7 321d 329d 3/3 fixed on 2023/07/22 07:09

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-....: (10499 ticks this GP) idle=1cd/1/0x4000000000000000 softirq=6194/6194 fqs=2506 
	(t=10500 jiffies g=6641 q=1351)
NMI backtrace for cpu 1
CPU: 1 PID: 19 Comm: ksoftirqd/1 Not tainted 5.16.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 nmi_cpu_backtrace.cold+0x47/0x144 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x1b3/0x230 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x25e/0x3f0 kernel/rcu/tree_stall.h:343
 print_cpu_stall kernel/rcu/tree_stall.h:627 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:711 [inline]
 rcu_pending kernel/rcu/tree.c:3878 [inline]
 rcu_sched_clock_irq.cold+0x5c/0x759 kernel/rcu/tree.c:2597
 update_process_times+0x16d/0x200 kernel/time/timer.c:1785
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1428
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x1c0/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:__sanitizer_cov_trace_const_cmp1+0x0/0x80 kernel/kcov.c:272
Code: 39 fe 72 1f 48 83 c2 01 4c 89 44 38 e8 48 c7 44 38 e0 06 00 00 00 48 89 4c 38 f0 4e 89 54 c8 20 48 89 10 c3 66 0f 1f 44 00 00 <53> 41 89 fb 41 89 f1 bf 03 00 00 00 65 48 8b 0c 25 40 70 02 00 48
RSP: 0018:ffffc90000d97b30 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88806c8340b0 RCX: 0000000000000100
RDX: ffff888011915700 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff88806c8340e8 R08: 0000000000000000 R09: 0000000000000001
R10: ffffffff875dae72 R11: 0000000000000000 R12: 0000000000000000
R13: ffff88807b06c300 R14: 0000000000000000 R15: 0000000000000000
 pie_calculate_probability+0x54e/0x7c0 net/sched/sch_pie.c:413
 fq_pie_timer+0x170/0x2a0 net/sched/sch_fq_pie.c:383
 call_timer_fn+0x1a5/0x6b0 kernel/time/timer.c:1421
 expire_timers kernel/time/timer.c:1466 [inline]
 __run_timers.part.0+0x67c/0xa30 kernel/time/timer.c:1734
 __run_timers kernel/time/timer.c:1715 [inline]
 run_timer_softirq+0xb3/0x1d0 kernel/time/timer.c:1747
 __do_softirq+0x29b/0x9c2 kernel/softirq.c:558
 run_ksoftirqd kernel/softirq.c:921 [inline]
 run_ksoftirqd+0x2d/0x60 kernel/softirq.c:913
 smpboot_thread_fn+0x645/0x9c0 kernel/smpboot.c:164
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
----------------
Code disassembly (best guess):
   0:	39 fe                	cmp    %edi,%esi
   2:	72 1f                	jb     0x23
   4:	48 83 c2 01          	add    $0x1,%rdx
   8:	4c 89 44 38 e8       	mov    %r8,-0x18(%rax,%rdi,1)
   d:	48 c7 44 38 e0 06 00 	movq   $0x6,-0x20(%rax,%rdi,1)
  14:	00 00
  16:	48 89 4c 38 f0       	mov    %rcx,-0x10(%rax,%rdi,1)
  1b:	4e 89 54 c8 20       	mov    %r10,0x20(%rax,%r9,8)
  20:	48 89 10             	mov    %rdx,(%rax)
  23:	c3                   	retq
  24:	66 0f 1f 44 00 00    	nopw   0x0(%rax,%rax,1)
* 2a:	53                   	push   %rbx <-- trapping instruction
  2b:	41 89 fb             	mov    %edi,%r11d
  2e:	41 89 f1             	mov    %esi,%r9d
  31:	bf 03 00 00 00       	mov    $0x3,%edi
  36:	65 48 8b 0c 25 40 70 	mov    %gs:0x27040,%rcx
  3d:	02 00
  3f:	48                   	rex.W

Crashes (35):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/12/31 06:18 upstream 74c78b4291b4 36bd2e48 .config console log report syz C ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/12/30 23:16 net-old f7397cd24c59 2e49f10d .config console log report syz C ci-upstream-net-this-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/12/08 22:43 upstream 2a987e65025e a4a2a501 .config console log report syz ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/12/08 22:58 net-old b5bd95d17102 a4a2a501 .config console log report syz ci-upstream-net-this-kasan-gce INFO: rcu detected stall in fq_pie_timer
2022/01/27 00:12 upstream 0280e3c58f92 2cbffd88 .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2022/01/14 06:32 upstream 455e73a07f6e b8d780ab .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/12/30 18:44 upstream eec4df26e24e 2e49f10d .config console log report info ci-upstream-kasan-gce-root INFO: rcu detected stall in fq_pie_timer
2021/12/19 23:52 upstream a76c3d035872 44068e19 .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/12/13 10:19 upstream 2585cf9dfaad 49ca1f59 .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/12/08 18:20 upstream 2a987e65025e a4a2a501 .config console log report info ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2021/12/08 18:00 upstream 2a987e65025e a4a2a501 .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/12/05 22:02 upstream 944207047ca4 a617004c .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/12/04 21:48 upstream bbef3c7a63d2 a617004c .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/11/27 07:21 upstream c5c17547b778 63eeac02 .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/11/20 00:28 upstream 4c388a8e740d 3a9d0024 .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/11/13 20:19 upstream a9b9669d9822 83f5c9b5 .config console log report info ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2021/11/12 16:16 upstream 5833291ab6de 83f5c9b5 .config console log report info ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2021/11/11 09:43 upstream debe436e77c7 75b04091 .config console log report info ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2021/11/11 09:01 upstream debe436e77c7 75b04091 .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/11/08 14:37 upstream 6b75d88fa81b d29682f1 .config console log report info ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in fq_pie_timer
2021/10/25 23:39 upstream c2b43854aad9 c1132b49 .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/09/05 15:41 upstream 49624efa65ac d236a457 .config console log report info ci-upstream-kasan-gce INFO: rcu detected stall in fq_pie_timer
2021/09/01 20:39 upstream 835d31d319d9 7eb7e152 .config console log report info ci-upstream-kasan-gce-root INFO: rcu detected stall in fq_pie_timer
2021/08/20 00:27 upstream f87d64319e6f b599f2fc .config console log report info ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in fq_pie_timer
2022/01/15 03:26 upstream 112450df61b7 723cfaf0 .config console log report info ci-upstream-kasan-gce-386 INFO: rcu detected stall in fq_pie_timer
2021/11/15 10:28 upstream fa55b7dcdc43 83f5c9b5 .config console log report info ci-upstream-kasan-gce-386 INFO: rcu detected stall in fq_pie_timer
2021/11/04 21:27 upstream 7ddb58cb0eca 4c1be0be .config console log report info ci-upstream-kasan-gce-386 INFO: rcu detected stall in fq_pie_timer
2021/06/22 19:21 upstream 0c18f29aae7c aba2b2fb .config console log report info ci-upstream-kasan-gce-386 INFO: rcu detected stall in fq_pie_timer
2021/11/06 07:22 net-old a46a5036e7d2 4c1be0be .config console log report info ci-upstream-net-this-kasan-gce INFO: rcu detected stall in fq_pie_timer
2022/01/19 08:32 linux-next 6f59bc242877 0620189b .config console log report info ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in fq_pie_timer
2021/12/31 00:37 linux-next ea586a076e8a 36bd2e48 .config console log report info ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in fq_pie_timer
2021/12/15 17:51 linux-next ea922272cbe5 572bcb40 .config console log report info ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in fq_pie_timer
2021/09/14 15:17 linux-next 815c5020b5ab 07e953c1 .config console log report info ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in fq_pie_timer
2021/12/07 15:12 upstream cd8c917a56f2 0230ba3e .config console log report info ci-upstream-kasan-gce BUG: soft lockup in fq_pie_timer
2021/11/19 14:23 upstream 4c388a8e740d 3a9d0024 .config console log report info ci-upstream-kasan-gce BUG: soft lockup in fq_pie_timer
* Struck through repros no longer work on HEAD.