syzbot


INFO: rcu detected stall in net_rx_action (2)

Status: closed as invalid on 2022/02/08 10:10
Reported-by: syzbot+@syzkaller.appspotmail.com
First crash: 369d, last: 369d
similar bugs (4):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in net_rx_action (3) 67 216d 216d 0/24 auto-closed as invalid on 2022/06/05 10:21
upstream INFO: rcu detected stall in net_rx_action 3 676d 684d 0/24 auto-closed as invalid on 2021/04/21 12:41
linux-4.19 BUG: soft lockup in net_rx_action (2) 1 52d 52d 0/1 upstream: reported on 2022/10/06 20:26
linux-4.19 BUG: soft lockup in net_rx_action 1 316d 316d 0/1 auto-closed as invalid on 2022/05/16 12:13

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	0-...!: (1 ticks this GP) idle=6fd/1/0x4000000000000000 softirq=19757/19757 fqs=44 
	(detected by 1, t=10502 jiffies, g=30241, q=2841)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 2735 Comm: bat_events Not tainted 5.16.0-rc1-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: bat_events batadv_iv_send_outstanding_bat_ogm_packet
RIP: 0010:hlock_class kernel/locking/lockdep.c:199 [inline]
RIP: 0010:mark_lock+0xf7/0x17b0 kernel/locking/lockdep.c:4583
Code: 66 81 e2 ff 1f 44 0f b7 f2 be 08 00 00 00 4c 89 f0 48 c1 f8 06 48 8d 3c c5 00 6a f7 8f e8 91 24 62 00 4c 0f a3 35 69 3a 9b 0e <0f> 83 03 01 00 00 4b 8d 04 76 48 c1 e0 06 48 05 20 6e f7 8f 48 8d
RSP: 0018:ffffc90000007768 EFLAGS: 00000047
RAX: 0000000000000001 RBX: 1ffff92000000ef4 RCX: ffffffff815c2f8f
RDX: fffffbfff1feed41 RSI: 0000000000000008 RDI: ffffffff8ff76a00
RBP: 0000000000000200 R08: 0000000000000000 R09: ffffffff8ff76a07
R10: fffffbfff1feed40 R11: 0000000000000000 R12: 0000000000000009
R13: ffff888023738ab0 R14: 0000000000000028 R15: ffff888023738ad0
FS:  0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f844d893fc0 CR3: 000000007e5cb000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 mark_usage kernel/locking/lockdep.c:4526 [inline]
 __lock_acquire+0x8a7/0x54a0 kernel/locking/lockdep.c:4981
 lock_acquire kernel/locking/lockdep.c:5637 [inline]
 lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5602
 rcu_lock_acquire include/linux/rcupdate.h:268 [inline]
 rcu_read_lock include/linux/rcupdate.h:688 [inline]
 advance_sched+0x4a5/0x9a0 net/sched/sch_taprio.c:767
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x609/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x40/0xc0 arch/x86/kernel/apic/apic.c:1097
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:skb_peek include/linux/skbuff.h:1843 [inline]
RIP: 0010:__skb_dequeue include/linux/skbuff.h:2149 [inline]
RIP: 0010:process_backlog+0x117/0x6c0 net/core/dev.c:6450
Code: 18 45 31 ed 49 bc 00 00 00 00 00 fc ff df 48 89 04 24 48 8d 83 50 fe ff ff 48 89 44 24 18 48 c1 e8 03 4c 01 e0 48 89 44 24 10 <4c> 8b 3c 24 49 c1 ef 03 4b 8d 04 27 48 89 44 24 08 eb 60 e8 11 1a
RSP: 0018:ffffc90000007dd8 EFLAGS: 00000206
RAX: 0000000000006a22 RBX: ffff8880b9c3ad90 RCX: 1ffffffff1ff9616
RDX: 0000000000000000 RSI: 0000000000000101 RDI: 0000000000000000
RBP: ffff8880b9c3ad50 R08: 0000000000000001 R09: ffffffff8ff76ae7
R10: 0000000000000001 R11: 0000000000000000 R12: dffffc0000000000
R13: 0000000000000000 R14: 0000000000000040 R15: ffff8880b9c3ad38
 __napi_poll+0xaf/0x440 net/core/dev.c:7020
 napi_poll net/core/dev.c:7087 [inline]
 net_rx_action+0x801/0xb40 net/core/dev.c:7174
 __do_softirq+0x29b/0x9c2 kernel/softirq.c:558
 do_softirq.part.0+0xde/0x130 kernel/softirq.c:459
 </IRQ>
 <TASK>
 do_softirq kernel/softirq.c:451 [inline]
 __local_bh_enable_ip+0x102/0x120 kernel/softirq.c:383
 local_bh_enable include/linux/bottom_half.h:33 [inline]
 rcu_read_unlock_bh include/linux/rcupdate.h:758 [inline]
 __dev_queue_xmit+0x196a/0x3630 net/core/dev.c:4253
 batadv_send_skb_packet+0x4a9/0x5f0 net/batman-adv/send.c:108
 batadv_iv_ogm_send_to_if net/batman-adv/bat_iv_ogm.c:393 [inline]
 batadv_iv_ogm_emit net/batman-adv/bat_iv_ogm.c:421 [inline]
 batadv_iv_send_outstanding_bat_ogm_packet+0x6d7/0x8e0 net/batman-adv/bat_iv_ogm.c:1701
 process_one_work+0x9b2/0x1690 kernel/workqueue.c:2298
 process_scheduled_works kernel/workqueue.c:2361 [inline]
 rescuer_thread+0x5a0/0xd20 kernel/workqueue.c:2552
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
rcu: rcu_preempt kthread starved for 9440 jiffies! g30241 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:28688 pid:   14 ppid:     2 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:4972 [inline]
 __schedule+0xa9a/0x4940 kernel/sched/core.c:6253
 schedule+0xd2/0x260 kernel/sched/core.c:6326
 schedule_timeout+0x14a/0x2a0 kernel/time/timer.c:1881
 rcu_gp_fqs_loop+0x186/0x810 kernel/rcu/tree.c:1955
 rcu_gp_kthread+0x1de/0x320 kernel/rcu/tree.c:2128
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
NMI backtrace for cpu 1
CPU: 1 PID: 13216 Comm: kworker/u4:17 Not tainted 5.16.0-rc1-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_unbound toggle_allocation_gate
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 nmi_cpu_backtrace.cold+0x47/0x144 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x1b3/0x230 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_check_gp_kthread_starvation.cold+0x1fb/0x200 kernel/rcu/tree_stall.h:481
 print_other_cpu_stall kernel/rcu/tree_stall.h:586 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:729 [inline]
 rcu_pending kernel/rcu/tree.c:3878 [inline]
 rcu_sched_clock_irq+0x2125/0x2200 kernel/rcu/tree.c:2597
 update_process_times+0x16d/0x200 kernel/time/timer.c:1785
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1421
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x1c0/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:rep_nop arch/x86/include/asm/vdso/processor.h:13 [inline]
RIP: 0010:cpu_relax arch/x86/include/asm/vdso/processor.h:18 [inline]
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0x44b/0xc20 kernel/smp.c:969
Code: e5 01 89 ee e8 c6 77 0b 00 85 ed 74 4d 48 b8 00 00 00 00 00 fc ff df 4d 89 f4 4c 89 f5 49 c1 ec 03 83 e5 07 49 01 c4 83 c5 03 <e8> 80 75 0b 00 f3 90 41 0f b6 04 24 40 38 c5 7c 08 84 c0 0f 85 33
RSP: 0018:ffffc9000452f9f8 EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffff8880b9c41d40 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffff88801d103a00 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000001
R10: ffffffff816c22b6 R11: 0000000000000000 R12: ffffed10173883a9
R13: 0000000000000000 R14: ffff8880b9c41d48 R15: 0000000000000001
 on_each_cpu_cond_mask+0x56/0xa0 kernel/smp.c:1135
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:1112 [inline]
 text_poke_bp_batch+0x1b3/0x560 arch/x86/kernel/alternative.c:1297
 text_poke_flush arch/x86/kernel/alternative.c:1451 [inline]
 text_poke_flush arch/x86/kernel/alternative.c:1448 [inline]
 text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1458
 arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
 jump_label_update+0x1d5/0x430 kernel/jump_label.c:830
 static_key_enable_cpuslocked+0x1b1/0x260 kernel/jump_label.c:177
 static_key_enable+0x16/0x20 kernel/jump_label.c:190
 toggle_allocation_gate mm/kfence/core.c:732 [inline]
 toggle_allocation_gate+0x100/0x390 mm/kfence/core.c:724
 process_one_work+0x9b2/0x1690 kernel/workqueue.c:2298
 worker_thread+0x658/0x11f0 kernel/workqueue.c:2445
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
----------------
Code disassembly (best guess):
   0:	66 81 e2 ff 1f       	and    $0x1fff,%dx
   5:	44 0f b7 f2          	movzwl %dx,%r14d
   9:	be 08 00 00 00       	mov    $0x8,%esi
   e:	4c 89 f0             	mov    %r14,%rax
  11:	48 c1 f8 06          	sar    $0x6,%rax
  15:	48 8d 3c c5 00 6a f7 	lea    -0x70089600(,%rax,8),%rdi
  1c:	8f
  1d:	e8 91 24 62 00       	callq  0x6224b3
  22:	4c 0f a3 35 69 3a 9b 	bt     %r14,0xe9b3a69(%rip)        # 0xe9b3a93
  29:	0e
* 2a:	0f 83 03 01 00 00    	jae    0x133 <-- trapping instruction
  30:	4b 8d 04 76          	lea    (%r14,%r14,2),%rax
  34:	48 c1 e0 06          	shl    $0x6,%rax
  38:	48 05 20 6e f7 8f    	add    $0xffffffff8ff76e20,%rax
  3e:	48                   	rex.W
  3f:	8d                   	.byte 0x8d

Crashes (1):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-upstream-net-this-kasan-gce 2021/11/23 23:50 net 7b1b62bc1e6a 545ab074 .config log report info INFO: rcu detected stall in net_rx_action
* Struck through repros no longer work on HEAD.