syzbot


INFO: rcu detected stall in tipc_conn_recv_work

Status: closed as invalid on 2022/02/08 10:10
Subsystems: kvm
[Documentation on labels]
First crash: 879d, last: 814d
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream BUG: soft lockup in tipc_conn_recv_work kvm 1 727d 723d 0/26 closed as invalid on 2022/05/02 06:41

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-...!: (1 GPs behind) idle=80f/1/0x4000000000000000 softirq=32965/32970 fqs=0 
	(t=10500 jiffies g=53437 q=43)
rcu: rcu_preempt kthread timer wakeup didn't happen for 10499 jiffies! g53437 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
rcu: 	Possible timer handling issue on cpu=0 timer-softirq=17463
rcu: rcu_preempt kthread starved for 10500 jiffies! g53437 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:I stack:28656 pid:   14 ppid:     2 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:4972 [inline]
 __schedule+0xa9a/0x4940 kernel/sched/core.c:6253
 schedule+0xd2/0x260 kernel/sched/core.c:6326
 schedule_timeout+0x14a/0x2a0 kernel/time/timer.c:1881
 rcu_gp_fqs_loop+0x186/0x810 kernel/rcu/tree.c:1955
 rcu_gp_kthread+0x1de/0x320 kernel/rcu/tree.c:2128
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 1088 Comm: kworker/u4:5 Not tainted 5.16.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: netns cleanup_net
RIP: 0010:radix_tree_lookup+0x2/0x20 lib/radix-tree.c:816
Code: 44 24 58 65 48 2b 04 25 28 00 00 00 75 11 48 83 c4 60 4c 89 e0 5b 5d 41 5c c3 49 89 c4 eb c8 e8 54 ea 44 05 0f 1f 40 00 41 54 <49> 89 f4 55 48 89 fd e8 52 78 78 fd 4c 89 e6 48 89 ef 5d 41 5c 31
RSP: 0018:ffffc90005277bd8 EFLAGS: 00000206
RAX: 0000000000000000 RBX: 0000000074d17f2e RCX: 0000000000000000
RDX: 1ffff1100d7d6a8a RSI: 0000000074d17f2e RDI: ffff88806beb5400
RBP: ffff88806beb5400 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff88ba7ee9 R11: 0000000000000000 R12: ffff88806beb5458
R13: ffffed100d7d6a93 R14: ffff88806c37b000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3fa3fa9004 CR3: 000000004fef5000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 tipc_topsrv_stop net/tipc/topsrv.c:691 [inline]
 tipc_topsrv_exit_net+0x20a/0x5d0 net/tipc/topsrv.c:715
 ops_exit_list+0xb0/0x160 net/core/net_namespace.c:168
 cleanup_net+0x4ea/0xb00 net/core/net_namespace.c:593
 process_one_work+0x9b2/0x1690 kernel/workqueue.c:2298
 worker_thread+0x658/0x11f0 kernel/workqueue.c:2445
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 1088 Comm: kworker/u4:5 Not tainted 5.16.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: netns cleanup_net
RIP: 0010:__sanitizer_cov_trace_const_cmp4+0x0/0x70 kernel/kcov.c:286
Code: b7 c9 48 83 c2 01 48 c7 44 30 e0 02 00 00 00 48 89 7c 30 e8 48 89 4c 30 f0 4c 89 54 d8 20 48 89 10 5b c3 0f 1f 80 00 00 00 00 <41> 89 f8 bf 03 00 00 00 4c 8b 14 24 89 f1 65 48 8b 34 25 40 70 02
RSP: 0018:ffffc90005277b80 EFLAGS: 00000246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff88801c2f5700 RSI: 0000000000000000 RDI: 000000000000003f
RBP: dffffc0000000000 R08: 0000000000000002 R09: 0000000000000000
R10: ffffffff83ff1e15 R11: 0000000000000000 R12: ffff88802a78de42
R13: 0000000000000000 R14: 0000000074f9f17b R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3fa3fa9004 CR3: 000000004fef5000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 shift_maxindex lib/radix-tree.c:211 [inline]
 node_maxindex lib/radix-tree.c:216 [inline]
 radix_tree_load_root lib/radix-tree.c:395 [inline]
 __radix_tree_lookup+0xb4/0x2a0 lib/radix-tree.c:756
 tipc_topsrv_stop net/tipc/topsrv.c:691 [inline]
 tipc_topsrv_exit_net+0x20a/0x5d0 net/tipc/topsrv.c:715
 ops_exit_list+0xb0/0x160 net/core/net_namespace.c:168
 cleanup_net+0x4ea/0xb00 net/core/net_namespace.c:593
 process_one_work+0x9b2/0x1690 kernel/workqueue.c:2298
 worker_thread+0x658/0x11f0 kernel/workqueue.c:2445
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
NMI backtrace for cpu 1
CPU: 1 PID: 18563 Comm: kworker/u4:8 Not tainted 5.16.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: tipc_rcv tipc_conn_recv_work
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 nmi_cpu_backtrace.cold+0x47/0x144 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x1b3/0x230 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x25e/0x3f0 kernel/rcu/tree_stall.h:343
 print_cpu_stall kernel/rcu/tree_stall.h:627 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:711 [inline]
 rcu_pending kernel/rcu/tree.c:3878 [inline]
 rcu_sched_clock_irq.cold+0x9d/0x746 kernel/rcu/tree.c:2597
 update_process_times+0x16d/0x200 kernel/time/timer.c:1785
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1421
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x1c0/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:kvm_wait+0x98/0x100 arch/x86/kernel/kvm.c:1001
Code: fa 83 e2 07 38 d0 7f 04 84 c0 75 63 0f b6 07 40 38 c6 74 35 48 83 c4 10 c3 c3 e8 73 56 4a 00 eb 07 0f 00 2d ea 8f 75 08 fb f4 <48> 83 c4 10 c3 89 74 24 0c 48 89 3c 24 e8 36 51 4a 00 8b 74 24 0c
RSP: 0018:ffffc90004c47b08 EFLAGS: 00000206
RAX: 000000000005533a RBX: 0000000000000000 RCX: 1ffffffff1ffafee
RDX: 0000000000000000 RSI: 0000000000000201 RDI: 0000000000000000
RBP: ffff88806beb5458 R08: 0000000000000001 R09: ffffffff8ff72b0f
R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
R13: ffffed100d7d6a8b R14: 0000000000000001 R15: ffff8880b9d3a880
 pv_wait arch/x86/include/asm/paravirt.h:603 [inline]
 pv_wait_head_or_lock kernel/locking/qspinlock_paravirt.h:470 [inline]
 __pv_queued_spin_lock_slowpath+0x8b8/0xb40 kernel/locking/qspinlock.c:508
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:591 [inline]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline]
 queued_spin_lock include/asm-generic/qspinlock.h:85 [inline]
 do_raw_spin_lock+0x200/0x2b0 kernel/locking/spinlock_debug.c:115
 spin_lock_bh include/linux/spinlock.h:354 [inline]
 tipc_conn_kref_release+0x47/0x380 net/tipc/topsrv.c:129
 kref_put include/linux/kref.h:65 [inline]
 conn_put net/tipc/topsrv.c:147 [inline]
 tipc_conn_recv_work+0x14a/0x190 net/tipc/topsrv.c:430
 process_one_work+0x9b2/0x1690 kernel/workqueue.c:2298
 worker_thread+0x658/0x11f0 kernel/workqueue.c:2445
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
----------------
Code disassembly (best guess):
   0:	44 24 58             	rex.R and $0x58,%al
   3:	65 48 2b 04 25 28 00 	sub    %gs:0x28,%rax
   a:	00 00
   c:	75 11                	jne    0x1f
   e:	48 83 c4 60          	add    $0x60,%rsp
  12:	4c 89 e0             	mov    %r12,%rax
  15:	5b                   	pop    %rbx
  16:	5d                   	pop    %rbp
  17:	41 5c                	pop    %r12
  19:	c3                   	retq
  1a:	49 89 c4             	mov    %rax,%r12
  1d:	eb c8                	jmp    0xffffffe7
  1f:	e8 54 ea 44 05       	callq  0x544ea78
  24:	0f 1f 40 00          	nopl   0x0(%rax)
  28:	41 54                	push   %r12
* 2a:	49 89 f4             	mov    %rsi,%r12 <-- trapping instruction
  2d:	55                   	push   %rbp
  2e:	48 89 fd             	mov    %rdi,%rbp
  31:	e8 52 78 78 fd       	callq  0xfd787888
  36:	4c 89 e6             	mov    %r12,%rsi
  39:	48 89 ef             	mov    %rbp,%rdi
  3c:	5d                   	pop    %rbp
  3d:	41 5c                	pop    %r12
  3f:	31                   	.byte 0x31

Crashes (4):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/12/01 00:34 net-old 1a59c9c55585 80270552 .config console log report info ci-upstream-net-this-kasan-gce INFO: rcu detected stall in tipc_conn_recv_work
2021/11/30 15:53 net-old 1a59c9c55585 80270552 .config console log report info ci-upstream-net-this-kasan-gce INFO: rcu detected stall in tipc_conn_recv_work
2021/11/27 04:06 net-old 32c54497545e 63eeac02 .config console log report info ci-upstream-net-this-kasan-gce INFO: rcu detected stall in tipc_conn_recv_work
2022/01/30 19:32 net-old 6449520391df 495e00c5 .config console log report info ci-upstream-net-this-kasan-gce BUG: soft lockup in tipc_conn_recv_work
* Struck through repros no longer work on HEAD.