rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 0-....: (1 GPs behind) idle=176/1/0x4000000000000002 softirq=16935/16938 fqs=5236
(detected by 1, t=10502 jiffies, g=23477, q=226285)
============================================
WARNING: possible recursive locking detected
5.12.0-rc4-syzkaller #0 Not tainted
--------------------------------------------
kworker/u4:2/29 is trying to acquire lock:
ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_dump_cpu_stacks+0xaf/0x2a7 kernel/rcu/tree_stall.h:336
but task is already holding lock:
ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: print_other_cpu_stall kernel/rcu/tree_stall.h:541 [inline]
ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: check_cpu_stall kernel/rcu/tree_stall.h:706 [inline]
ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_pending kernel/rcu/tree.c:3830 [inline]
ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_sched_clock_irq.cold+0x2b9/0x11dd kernel/rcu/tree.c:2650
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(rcu_node_0);
lock(rcu_node_0);
*** DEADLOCK ***
May be due to missing lock nesting notation
9 locks held by kworker/u4:2/29:
#0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
#0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: atomic64_set include/asm-generic/atomic-instrumented.h:856 [inline]
#0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: atomic_long_set include/asm-generic/atomic-long.h:41 [inline]
#0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:616 [inline]
#0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline]
#0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: process_one_work+0x871/0x1600 kernel/workqueue.c:2246
#1: ffffc90000e4fda8 ((work_completion)(&(&cp->cp_conn_w)->work)){+.+.}-{0:0}, at: process_one_work+0x8a5/0x1600 kernel/workqueue.c:2250
#2: ffff888023ddc088 (&tc->t_conn_path_lock){+.+.}-{3:3}, at: rds_tcp_conn_path_connect+0x174/0x880 net/rds/tcp_connect.c:107
#3: ffff8880142bb020 (k-sk_lock-AF_INET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1606 [inline]
#3: ffff8880142bb020 (k-sk_lock-AF_INET){+.+.}-{0:0}, at: inet_stream_connect+0x3f/0xa0 net/ipv4/af_inet.c:727
#4: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: read_pnet include/net/net_namespace.h:322 [inline]
#4: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: sock_net include/net/sock.h:2560 [inline]
#4: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __ip_queue_xmit+0x3f/0x1a20 net/ipv4/ip_output.c:458
#5: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __skb_unlink include/linux/skbuff.h:2089 [inline]
#5: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __skb_dequeue include/linux/skbuff.h:2104 [inline]
#5: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: process_backlog+0x1c1/0x6c0 net/core/dev.c:6411
#6: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __skb_pull include/linux/skbuff.h:2320 [inline]
#6: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: ip_local_deliver_finish+0x124/0x370 net/ipv4/ip_input.c:228
#7: ffff8880144c3de0 (k-slock-AF_INET6){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
#7: ffff8880144c3de0 (k-slock-AF_INET6){+.-.}-{2:2}, at: sk_clone_lock+0x296/0x1070 net/core/sock.c:1913
#8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: print_other_cpu_stall kernel/rcu/tree_stall.h:541 [inline]
#8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: check_cpu_stall kernel/rcu/tree_stall.h:706 [inline]
#8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_pending kernel/rcu/tree.c:3830 [inline]
#8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_sched_clock_irq.cold+0x2b9/0x11dd kernel/rcu/tree.c:2650
stack backtrace:
CPU: 1 PID: 29 Comm: kworker/u4:2 Not tainted 5.12.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: krdsd rds_connect_worker
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:79 [inline]
dump_stack+0x141/0x1d7 lib/dump_stack.c:120
print_deadlock_bug kernel/locking/lockdep.c:2829 [inline]
check_deadlock kernel/locking/lockdep.c:2872 [inline]
validate_chain kernel/locking/lockdep.c:3661 [inline]
__lock_acquire.cold+0x14c/0x3b4 kernel/locking/lockdep.c:4900
lock_acquire kernel/locking/lockdep.c:5510 [inline]
lock_acquire+0x1ab/0x740 kernel/locking/lockdep.c:5475
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
rcu_dump_cpu_stacks+0xaf/0x2a7 kernel/rcu/tree_stall.h:336
print_other_cpu_stall kernel/rcu/tree_stall.h:559 [inline]
check_cpu_stall kernel/rcu/tree_stall.h:706 [inline]
rcu_pending kernel/rcu/tree.c:3830 [inline]
rcu_sched_clock_irq.cold+0x7e1/0x11dd kernel/rcu/tree.c:2650
update_process_times+0x16d/0x200 kernel/time/timer.c:1796
tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1369
__run_hrtimer kernel/time/hrtimer.c:1537 [inline]
__hrtimer_run_queues+0x1c0/0xe40 kernel/time/hrtimer.c:1601
hrtimer_interrupt+0x330/0xa00 kernel/time/hrtimer.c:1663
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
__sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
sysvec_apic_timer_interrupt+0x40/0xc0 arch/x86/kernel/apic/apic.c:1100
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:632
RIP: 0010:__orc_find+0x47/0xf0 arch/x86/kernel/unwind_orc.c:50
Code: 4c 39 e7 49 89 fd 0f 87 9a 00 00 00 49 bf 00 00 00 00 00 fc ff df 49 89 fe 48 89 fd eb 0c 48 8d 6b 04 49 89 de 49 39 ec 72 4d <4c> 89 e0 48 29 e8 48 89 c2 48 c1 e8 3f 48 c1 fa 02 48 01 d0 48 d1
RSP: 0018:ffffc90000dbffb0 EFLAGS: 00000246
RAX: ffffffff8783b9b5 RBX: ffffffff8e24280c RCX: ffffffff8783ba2c
RDX: 0000000000000000 RSI: ffffffff8eb85f7e RDI: ffffffff8e24280c
RBP: ffffffff8e242810 R08: 0000000000000000 R09: ffffffff8eb85f7e
R10: fffff520001b8025 R11: 0000000000084087 R12: ffffffff8e242810
R13: ffffffff8e24280c R14: ffffffff8e24280c R15: dffffc0000000000
orc_find arch/x86/kernel/unwind_orc.c:173 [inline]
unwind_next_frame+0x32a/0x1ce0 arch/x86/kernel/unwind_orc.c:443
arch_stack_walk+0x7d/0xe0 arch/x86/kernel/stacktrace.c:25
stack_trace_save+0x8c/0xc0 kernel/stacktrace.c:121
kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38
kasan_set_track mm/kasan/common.c:46 [inline]
set_alloc_info mm/kasan/common.c:427 [inline]
__kasan_slab_alloc+0x75/0x90 mm/kasan/common.c:460
kasan_slab_alloc include/linux/kasan.h:223 [inline]
slab_post_alloc_hook mm/slab.h:516 [inline]
slab_alloc_node mm/slub.c:2907 [inline]
slab_alloc mm/slub.c:2915 [inline]
kmem_cache_alloc+0x155/0x370 mm/slub.c:2920
kmem_cache_zalloc include/linux/slab.h:674 [inline]
fill_pool+0x264/0x5c0 lib/debugobjects.c:171
__debug_object_init+0x7a/0xd10 lib/debugobjects.c:560
debug_hrtimer_init kernel/time/hrtimer.c:414 [inline]
debug_init kernel/time/hrtimer.c:468 [inline]
hrtimer_init+0x23/0x2b0 kernel/time/hrtimer.c:1444
tcp_init_xmit_timers+0x7b/0xc0 net/ipv4/tcp_timer.c:795
tcp_create_openreq_child+0x6b9/0x16c0 net/ipv4/tcp_minisocks.c:502
tcp_v4_syn_recv_sock+0x10c/0x1470 net/ipv4/tcp_ipv4.c:1541
tcp_v6_syn_recv_sock+0x157a/0x2350 net/ipv6/tcp_ipv6.c:1238
tcp_check_req+0x616/0x18c0 net/ipv4/tcp_minisocks.c:772
tcp_v4_rcv+0x221a/0x3780 net/ipv4/tcp_ipv4.c:2016
ip_protocol_deliver_rcu+0x5c/0x8a0 net/ipv4/ip_input.c:204
ip_local_deliver_finish+0x20a/0x370 net/ipv4/ip_input.c:231
NF_HOOK include/linux/netfilter.h:301 [inline]
NF_HOOK include/linux/netfilter.h:295 [inline]
ip_local_deliver+0x1b3/0x200 net/ipv4/ip_input.c:252
dst_input include/net/dst.h:458 [inline]
ip_rcv_finish+0x1da/0x2f0 net/ipv4/ip_input.c:429
NF_HOOK include/linux/netfilter.h:301 [inline]
NF_HOOK include/linux/netfilter.h:295 [inline]
ip_rcv+0xaa/0xd0 net/ipv4/ip_input.c:540
__netif_receive_skb_one_core+0x114/0x180 net/core/dev.c:5436
__netif_receive_skb+0x27/0x1c0 net/core/dev.c:5550
process_backlog+0x232/0x6c0 net/core/dev.c:6413
__napi_poll+0xaf/0x440 net/core/dev.c:6961
napi_poll net/core/dev.c:7028 [inline]
net_rx_action+0x801/0xb40 net/core/dev.c:7114
__do_softirq+0x29b/0x9f6 kernel/softirq.c:345
do_softirq.part.0+0xd9/0x130 kernel/softirq.c:248
</IRQ>
do_softirq kernel/softirq.c:240 [inline]
__local_bh_enable_ip+0x102/0x120 kernel/softirq.c:198
local_bh_enable include/linux/bottom_half.h:32 [inline]
rcu_read_unlock_bh include/linux/rcupdate.h:745 [inline]
ip_finish_output2+0x88b/0x21f0 net/ipv4/ip_output.c:231
__ip_finish_output net/ipv4/ip_output.c:308 [inline]
__ip_finish_output+0x396/0x640 net/ipv4/ip_output.c:290
ip_finish_output+0x35/0x200 net/ipv4/ip_output.c:318
NF_HOOK_COND include/linux/netfilter.h:290 [inline]
ip_output+0x196/0x310 net/ipv4/ip_output.c:432
dst_output include/net/dst.h:448 [inline]
ip_local_out net/ipv4/ip_output.c:126 [inline]
__ip_queue_xmit+0x905/0x1a20 net/ipv4/ip_output.c:533
__tcp_transmit_skb+0x1889/0x38f0 net/ipv4/tcp_output.c:1405
__tcp_send_ack.part.0+0x3d6/0x7a0 net/ipv4/tcp_output.c:3970
__tcp_send_ack net/ipv4/tcp_output.c:3976 [inline]
tcp_send_ack+0x7d/0xa0 net/ipv4/tcp_output.c:3976
tcp_rcv_synsent_state_process net/ipv4/tcp_input.c:6186 [inline]
tcp_rcv_state_process+0x3858/0x4bb0 net/ipv4/tcp_input.c:6355
tcp_v4_do_rcv+0x320/0x870 net/ipv4/tcp_ipv4.c:1716
sk_backlog_rcv include/net/sock.h:1016 [inline]
__release_sock+0x134/0x3b0 net/core/sock.c:2556
release_sock+0x54/0x1b0 net/core/sock.c:3080
inet_stream_connect+0x76/0xa0 net/ipv4/af_inet.c:729
rds_tcp_conn_path_connect+0x61c/0x880 net/rds/tcp_connect.c:172
rds_connect_worker+0x1a5/0x2c0 net/rds/threads.c:176
process_one_work+0x98d/0x1600 kernel/workqueue.c:2275
worker_thread+0x64c/0x1120 kernel/workqueue.c:2421
kthread+0x3b1/0x4a0 kernel/kthread.c:292
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294