syzbot

 sign-in | mailing list | source | docs
🐞 Open [968] ≡ Subsystems 🐞 Fixed [4556] 🐞 Invalid [10489] 📈 Kernel Health 📈 Bug Lifetimes 📈 Fuzzing 📈 Crashes 💬 Send us feedback

INFO: rcu detected stall in net_tx_action

Status: upstream: reported C repro on 2021/07/01 15:50
Labels: net (incorrect?)
Reported-by: syzbot+3ba0493d523d007b3819@syzkaller.appspotmail.com
First crash: 701d, last: 10h19m

Cause bisection: introduced by (bisect log) [merge commit]:
commit 08b0109eea36d71f87b863045d91bbcee98758bd
Author: Kalle Valo <kvalo@codeaurora.org>
Date: Mon Oct 1 15:49:48 2018 +0000

  Merge tag 'iwlwifi-next-for-kalle-2018-09-28' of git://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/iwlwifi-next

Crash: general protection fault in batadv_iv_ogm_queue_add (log)
Repro: C syz .config

Fix bisection: fixed by (bisect log) :
commit 0a9a25ca78437b39e691bcc3dc8240455b803d8d
Author: Ming Lei <ming.lei@redhat.com>
Date: Fri Mar 18 13:01:43 2022 +0000

  block: let blkcg_gq grab request queue's refcnt

▶ ▼ Discussions (1)
Title Replies (including bot) Last reply
[syzbot] INFO: rcu detected stall in net_tx_action 1 (3) 2022/07/27 16:00
▶ ▼ Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 BUG: soft lockup in net_tx_action 1 626d 626d 0/1 auto-closed as invalid on 2022/01/08 10:24
▶ ▼ Last patch testing requests (22)
Created Duration User Patch Repo Result
2023/05/12 23:51 15m retest repro net-next report log
2023/05/12 23:51 18m retest repro net report log
2023/05/12 23:51 16m retest repro net-next report log
2023/03/01 19:32 0m retest repro net-next-old error
2023/03/01 18:32 0m retest repro net-next-old error
2023/03/01 18:32 0m retest repro net-old error
2022/11/21 06:30 21m retest repro net-next-old report log
2022/11/21 06:30 15m retest repro net-old report log
2022/07/31 02:02 18m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git f80e2148 OK log
2022/07/30 22:33 10m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git f80e2148 report log
2022/07/30 11:44 10m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git f80e2148 report log
2022/07/30 09:47 16m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git f80e2148 error
2022/07/28 09:56 18m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git f80e2148 OK log
2022/07/28 08:13 18m hdanton@sina.com upstream OK log
2022/07/28 07:33 12m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git f80e21489590 report log
2022/07/28 04:29 13m hdanton@sina.com https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git f80e21489590 report log
2022/07/28 03:26 4m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git d6765985 error
2022/07/28 02:46 12m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git d6765985 report log
2022/07/28 00:13 13m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git d6765985 report log
2022/07/27 13:28 14m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git d6765985 report log
2022/07/27 13:00 14m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git d6765985 report log
2022/07/27 12:01 11m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git d6765985 report log

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-...!: (11 ticks this GP) idle=eae/0/0x3 softirq=11934/11934 fqs=0 
	(t=13378 jiffies g=11625 q=43)
rcu: rcu_preempt kthread starved for 13378 jiffies! g11625 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:28800 pid:   14 ppid:     2 flags:0x00004000
Call Trace:
 context_switch kernel/sched/core.c:4339 [inline]
 __schedule+0x916/0x23e0 kernel/sched/core.c:5147
 schedule+0xcf/0x270 kernel/sched/core.c:5226
 schedule_timeout+0x14a/0x250 kernel/time/timer.c:1892
 rcu_gp_fqs_loop kernel/rcu/tree.c:2004 [inline]
 rcu_gp_kthread+0xd07/0x2300 kernel/rcu/tree.c:2177
 kthread+0x3b1/0x4a0 kernel/kthread.c:313
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
rcu: Stack dump where RCU GP kthread last ran:
NMI backtrace for cpu 1
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.13.0-rc6-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x141/0x1d7 lib/dump_stack.c:120
 nmi_cpu_backtrace.cold+0x44/0xd7 lib/nmi_backtrace.c:105
 nmi_trigger_cpumask_backtrace+0x1b3/0x230 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_check_gp_kthread_starvation.cold+0x1cc/0x1d1 kernel/rcu/tree_stall.h:478
 print_cpu_stall kernel/rcu/tree_stall.h:622 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:699 [inline]
 rcu_pending kernel/rcu/tree.c:3911 [inline]
 rcu_sched_clock_irq.cold+0x3ec/0x74b kernel/rcu/tree.c:2649
 update_process_times+0x16d/0x200 kernel/time/timer.c:1796
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1374
 __run_hrtimer kernel/time/hrtimer.c:1537 [inline]
 __hrtimer_run_queues+0x1c0/0xe40 kernel/time/hrtimer.c:1601
 hrtimer_interrupt+0x330/0xa00 kernel/time/hrtimer.c:1663
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
 sysvec_apic_timer_interrupt+0x40/0xc0 arch/x86/kernel/apic/apic.c:1100
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:647
RIP: 0010:arch_safe_halt arch/x86/include/asm/irqflags.h:90 [inline]
RIP: 0010:kvm_wait arch/x86/kernel/kvm.c:888 [inline]
RIP: 0010:kvm_wait+0xb2/0x100 arch/x86/kernel/kvm.c:871
Code: 89 74 24 0c 48 89 3c 24 e8 2b 2e 48 00 8b 74 24 0c 48 8b 3c 24 eb 82 e8 4c 33 48 00 e9 07 00 00 00 0f 00 2d 90 10 36 08 fb f4 <eb> 98 e9 07 00 00 00 0f 00 2d 80 10 36 08 f4 eb bf 89 74 24 0c 48
RSP: 0018:ffffc90000dc0d98 EFLAGS: 00000202
RAX: 0000000000097204 RBX: 0000000000000000 RCX: 1ffffffff204f312
RDX: 0000000000000000 RSI: 0000000000000102 RDI: 0000000000000000
RBP: ffff8880312710f0 R08: 0000000000000001 R09: ffffffff90228977
R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
R13: ffffed100624e21e R14: 0000000000000001 R15: ffff8880b9d36400
 pv_wait arch/x86/include/asm/paravirt.h:597 [inline]
 pv_wait_head_or_lock kernel/locking/qspinlock_paravirt.h:470 [inline]
 __pv_queued_spin_lock_slowpath+0x8b8/0xb40 kernel/locking/qspinlock.c:508
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:585 [inline]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline]
 queued_spin_lock include/asm-generic/qspinlock.h:85 [inline]
 do_raw_spin_lock+0x200/0x2b0 kernel/locking/spinlock_debug.c:113
 spin_lock include/linux/spinlock.h:354 [inline]
 net_tx_action+0x437/0xe10 net/core/dev.c:5044
 __do_softirq+0x29b/0x9f6 kernel/softirq.c:559
 invoke_softirq kernel/softirq.c:433 [inline]
 __irq_exit_rcu+0x136/0x200 kernel/softirq.c:637
 irq_exit_rcu+0x5/0x20 kernel/softirq.c:649
 sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1100
 </IRQ>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:647
RIP: 0010:native_save_fl arch/x86/include/asm/irqflags.h:29 [inline]
RIP: 0010:arch_local_save_flags arch/x86/include/asm/irqflags.h:70 [inline]
RIP: 0010:arch_irqs_disabled arch/x86/include/asm/irqflags.h:132 [inline]
RIP: 0010:acpi_safe_halt drivers/acpi/processor_idle.c:109 [inline]
RIP: 0010:acpi_idle_do_entry+0x1c9/0x250 drivers/acpi/processor_idle.c:513
Code: 9d 1f 5b f8 84 db 75 ac e8 e4 18 5b f8 e8 ff 25 61 f8 e9 0c 00 00 00 e8 d5 18 5b f8 0f 00 2d 3e b7 b4 00 e8 c9 18 5b f8 fb f4 <9c> 5b 81 e3 00 02 00 00 fa 31 ff 48 89 de e8 d4 20 5b f8 48 85 db
RSP: 0018:ffffc90000d57d18 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8880123d54c0 RSI: ffffffff8919c327 RDI: 0000000000000000
RBP: ffff8880190bb064 R08: 0000000000000001 R09: 0000000000000001
R10: ffffffff817ae948 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8880190bb000 R14: ffff8880190bb064 R15: ffff88801c57c804
 acpi_idle_enter+0x361/0x500 drivers/acpi/processor_idle.c:648
 cpuidle_enter_state+0x1b1/0xc80 drivers/cpuidle/cpuidle.c:237
 cpuidle_enter+0x4a/0xa0 drivers/cpuidle/cpuidle.c:351
 call_cpuidle kernel/sched/idle.c:158 [inline]
 cpuidle_idle_call kernel/sched/idle.c:239 [inline]
 do_idle+0x3e8/0x590 kernel/sched/idle.c:306
 cpu_startup_entry+0x14/0x20 kernel/sched/idle.c:403
 start_secondary+0x274/0x350 arch/x86/kernel/smpboot.c:272
 secondary_startup_64_no_verify+0xb0/0xbb
NMI backtrace for cpu 1
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.13.0-rc6-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x141/0x1d7 lib/dump_stack.c:120
 nmi_cpu_backtrace.cold+0x44/0xd7 lib/nmi_backtrace.c:105
 nmi_trigger_cpumask_backtrace+0x1b3/0x230 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x25e/0x3f0 kernel/rcu/tree_stall.h:341
 print_cpu_stall kernel/rcu/tree_stall.h:624 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:699 [inline]
 rcu_pending kernel/rcu/tree.c:3911 [inline]
 rcu_sched_clock_irq.cold+0x3f1/0x74b kernel/rcu/tree.c:2649
 update_process_times+0x16d/0x200 kernel/time/timer.c:1796
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1374
 __run_hrtimer kernel/time/hrtimer.c:1537 [inline]
 __hrtimer_run_queues+0x1c0/0xe40 kernel/time/hrtimer.c:1601
 hrtimer_interrupt+0x330/0xa00 kernel/time/hrtimer.c:1663
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
 sysvec_apic_timer_interrupt+0x40/0xc0 arch/x86/kernel/apic/apic.c:1100
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:647
RIP: 0010:arch_safe_halt arch/x86/include/asm/irqflags.h:90 [inline]
RIP: 0010:kvm_wait arch/x86/kernel/kvm.c:888 [inline]
RIP: 0010:kvm_wait+0xb2/0x100 arch/x86/kernel/kvm.c:871
Code: 89 74 24 0c 48 89 3c 24 e8 2b 2e 48 00 8b 74 24 0c 48 8b 3c 24 eb 82 e8 4c 33 48 00 e9 07 00 00 00 0f 00 2d 90 10 36 08 fb f4 <eb> 98 e9 07 00 00 00 0f 00 2d 80 10 36 08 f4 eb bf 89 74 24 0c 48
RSP: 0018:ffffc90000dc0d98 EFLAGS: 00000202
RAX: 0000000000097204 RBX: 0000000000000000 RCX: 1ffffffff204f312
RDX: 0000000000000000 RSI: 0000000000000102 RDI: 0000000000000000
RBP: ffff8880312710f0 R08: 0000000000000001 R09: ffffffff90228977
R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
R13: ffffed100624e21e R14: 0000000000000001 R15: ffff8880b9d36400
 pv_wait arch/x86/include/asm/paravirt.h:597 [inline]
 pv_wait_head_or_lock kernel/locking/qspinlock_paravirt.h:470 [inline]
 __pv_queued_spin_lock_slowpath+0x8b8/0xb40 kernel/locking/qspinlock.c:508
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:585 [inline]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline]
 queued_spin_lock include/asm-generic/qspinlock.h:85 [inline]
 do_raw_spin_lock+0x200/0x2b0 kernel/locking/spinlock_debug.c:113
 spin_lock include/linux/spinlock.h:354 [inline]
 net_tx_action+0x437/0xe10 net/core/dev.c:5044
 __do_softirq+0x29b/0x9f6 kernel/softirq.c:559
 invoke_softirq kernel/softirq.c:433 [inline]
 __irq_exit_rcu+0x136/0x200 kernel/softirq.c:637
 irq_exit_rcu+0x5/0x20 kernel/softirq.c:649
 sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1100
 </IRQ>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:647
RIP: 0010:native_save_fl arch/x86/include/asm/irqflags.h:29 [inline]
RIP: 0010:arch_local_save_flags arch/x86/include/asm/irqflags.h:70 [inline]
RIP: 0010:arch_irqs_disabled arch/x86/include/asm/irqflags.h:132 [inline]
RIP: 0010:acpi_safe_halt drivers/acpi/processor_idle.c:109 [inline]
RIP: 0010:acpi_idle_do_entry+0x1c9/0x250 drivers/acpi/processor_idle.c:513
Code: 9d 1f 5b f8 84 db 75 ac e8 e4 18 5b f8 e8 ff 25 61 f8 e9 0c 00 00 00 e8 d5 18 5b f8 0f 00 2d 3e b7 b4 00 e8 c9 18 5b f8 fb f4 <9c> 5b 81 e3 00 02 00 00 fa 31 ff 48 89 de e8 d4 20 5b f8 48 85 db
RSP: 0018:ffffc90000d57d18 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8880123d54c0 RSI: ffffffff8919c327 RDI: 0000000000000000
RBP: ffff8880190bb064 R08: 0000000000000001 R09: 0000000000000001
R10: ffffffff817ae948 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8880190bb000 R14: ffff8880190bb064 R15: ffff88801c57c804
 acpi_idle_enter+0x361/0x500 drivers/acpi/processor_idle.c:648
 cpuidle_enter_state+0x1b1/0xc80 drivers/cpuidle/cpuidle.c:237
 cpuidle_enter+0x4a/0xa0 drivers/cpuidle/cpuidle.c:351
 call_cpuidle kernel/sched/idle.c:158 [inline]
 cpuidle_idle_call kernel/sched/idle.c:239 [inline]
 do_idle+0x3e8/0x590 kernel/sched/idle.c:306
 cpu_startup_entry+0x14/0x20 kernel/sched/idle.c:403
 start_secondary+0x274/0x350 arch/x86/kernel/smpboot.c:272
 secondary_startup_64_no_verify+0xb0/0xbb

Crashes (7):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets Manager Title
2021/06/27 15:39 net-old d6765985a42a 9d2ab5df .config console log report syz C ci-upstream-net-this-kasan-gce INFO: rcu detected stall in net_tx_action
2023/05/29 22:52 net-next 75455b906d82 cf184559 .config strace log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: rcu detected stall in net_tx_action
2022/08/13 06:26 net-next-old 7ebfc85e2cd7 8dfcaa3d .config strace log report syz C ci-upstream-net-kasan-gce INFO: rcu detected stall in net_tx_action
2023/05/29 23:45 net 9b9e46aa0727 cf184559 .config console log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce BUG: soft lockup in net_tx_action
2022/01/02 16:20 net-next-old c95e078069bf e1768e9c .config console log report syz C ci-upstream-net-kasan-gce BUG: soft lockup in net_tx_action
2022/08/13 03:15 net-next-old 7ebfc85e2cd7 8dfcaa3d .config console log report info ci-upstream-net-kasan-gce INFO: rcu detected stall in net_tx_action
2022/04/25 22:24 linux-next e7d6987e09a3 152baedd .config console log report info ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in net_tx_action
* Struck through repros no longer work on HEAD.