rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: rcu: 0-....: (1 GPs behind) idle=176/1/0x4000000000000002 softirq=16935/16938 fqs=5236 (detected by 1, t=10502 jiffies, g=23477, q=226285) ============================================ WARNING: possible recursive locking detected 5.12.0-rc4-syzkaller #0 Not tainted -------------------------------------------- kworker/u4:2/29 is trying to acquire lock: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_dump_cpu_stacks+0xaf/0x2a7 kernel/rcu/tree_stall.h:336 but task is already holding lock: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: print_other_cpu_stall kernel/rcu/tree_stall.h:541 [inline] ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: check_cpu_stall kernel/rcu/tree_stall.h:706 [inline] ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_pending kernel/rcu/tree.c:3830 [inline] ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_sched_clock_irq.cold+0x2b9/0x11dd kernel/rcu/tree.c:2650 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(rcu_node_0); lock(rcu_node_0); *** DEADLOCK *** May be due to missing lock nesting notation 9 locks held by kworker/u4:2/29: #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: atomic64_set include/asm-generic/atomic-instrumented.h:856 [inline] #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: atomic_long_set include/asm-generic/atomic-long.h:41 [inline] #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:616 [inline] #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline] #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: process_one_work+0x871/0x1600 kernel/workqueue.c:2246 #1: ffffc90000e4fda8 ((work_completion)(&(&cp->cp_conn_w)->work)){+.+.}-{0:0}, at: process_one_work+0x8a5/0x1600 kernel/workqueue.c:2250 #2: ffff888023ddc088 (&tc->t_conn_path_lock){+.+.}-{3:3}, at: rds_tcp_conn_path_connect+0x174/0x880 net/rds/tcp_connect.c:107 #3: ffff8880142bb020 (k-sk_lock-AF_INET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1606 [inline] #3: ffff8880142bb020 (k-sk_lock-AF_INET){+.+.}-{0:0}, at: inet_stream_connect+0x3f/0xa0 net/ipv4/af_inet.c:727 #4: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: read_pnet include/net/net_namespace.h:322 [inline] #4: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: sock_net include/net/sock.h:2560 [inline] #4: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __ip_queue_xmit+0x3f/0x1a20 net/ipv4/ip_output.c:458 #5: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __skb_unlink include/linux/skbuff.h:2089 [inline] #5: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __skb_dequeue include/linux/skbuff.h:2104 [inline] #5: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: process_backlog+0x1c1/0x6c0 net/core/dev.c:6411 #6: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __skb_pull include/linux/skbuff.h:2320 [inline] #6: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: ip_local_deliver_finish+0x124/0x370 net/ipv4/ip_input.c:228 #7: ffff8880144c3de0 (k-slock-AF_INET6){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline] #7: ffff8880144c3de0 (k-slock-AF_INET6){+.-.}-{2:2}, at: sk_clone_lock+0x296/0x1070 net/core/sock.c:1913 #8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: print_other_cpu_stall kernel/rcu/tree_stall.h:541 [inline] #8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: check_cpu_stall kernel/rcu/tree_stall.h:706 [inline] #8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_pending kernel/rcu/tree.c:3830 [inline] #8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_sched_clock_irq.cold+0x2b9/0x11dd kernel/rcu/tree.c:2650 stack backtrace: CPU: 1 PID: 29 Comm: kworker/u4:2 Not tainted 5.12.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: krdsd rds_connect_worker Call Trace: __dump_stack lib/dump_stack.c:79 [inline] dump_stack+0x141/0x1d7 lib/dump_stack.c:120 print_deadlock_bug kernel/locking/lockdep.c:2829 [inline] check_deadlock kernel/locking/lockdep.c:2872 [inline] validate_chain kernel/locking/lockdep.c:3661 [inline] __lock_acquire.cold+0x14c/0x3b4 kernel/locking/lockdep.c:4900 lock_acquire kernel/locking/lockdep.c:5510 [inline] lock_acquire+0x1ab/0x740 kernel/locking/lockdep.c:5475 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline] _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159 rcu_dump_cpu_stacks+0xaf/0x2a7 kernel/rcu/tree_stall.h:336 print_other_cpu_stall kernel/rcu/tree_stall.h:559 [inline] check_cpu_stall kernel/rcu/tree_stall.h:706 [inline] rcu_pending kernel/rcu/tree.c:3830 [inline] rcu_sched_clock_irq.cold+0x7e1/0x11dd kernel/rcu/tree.c:2650 update_process_times+0x16d/0x200 kernel/time/timer.c:1796 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1369 __run_hrtimer kernel/time/hrtimer.c:1537 [inline] __hrtimer_run_queues+0x1c0/0xe40 kernel/time/hrtimer.c:1601 hrtimer_interrupt+0x330/0xa00 kernel/time/hrtimer.c:1663 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline] __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106 sysvec_apic_timer_interrupt+0x40/0xc0 arch/x86/kernel/apic/apic.c:1100 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:632 RIP: 0010:__orc_find+0x47/0xf0 arch/x86/kernel/unwind_orc.c:50 Code: 4c 39 e7 49 89 fd 0f 87 9a 00 00 00 49 bf 00 00 00 00 00 fc ff df 49 89 fe 48 89 fd eb 0c 48 8d 6b 04 49 89 de 49 39 ec 72 4d <4c> 89 e0 48 29 e8 48 89 c2 48 c1 e8 3f 48 c1 fa 02 48 01 d0 48 d1 RSP: 0018:ffffc90000dbffb0 EFLAGS: 00000246 RAX: ffffffff8783b9b5 RBX: ffffffff8e24280c RCX: ffffffff8783ba2c RDX: 0000000000000000 RSI: ffffffff8eb85f7e RDI: ffffffff8e24280c RBP: ffffffff8e242810 R08: 0000000000000000 R09: ffffffff8eb85f7e R10: fffff520001b8025 R11: 0000000000084087 R12: ffffffff8e242810 R13: ffffffff8e24280c R14: ffffffff8e24280c R15: dffffc0000000000 orc_find arch/x86/kernel/unwind_orc.c:173 [inline] unwind_next_frame+0x32a/0x1ce0 arch/x86/kernel/unwind_orc.c:443 arch_stack_walk+0x7d/0xe0 arch/x86/kernel/stacktrace.c:25 stack_trace_save+0x8c/0xc0 kernel/stacktrace.c:121 kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38 kasan_set_track mm/kasan/common.c:46 [inline] set_alloc_info mm/kasan/common.c:427 [inline] __kasan_slab_alloc+0x75/0x90 mm/kasan/common.c:460 kasan_slab_alloc include/linux/kasan.h:223 [inline] slab_post_alloc_hook mm/slab.h:516 [inline] slab_alloc_node mm/slub.c:2907 [inline] slab_alloc mm/slub.c:2915 [inline] kmem_cache_alloc+0x155/0x370 mm/slub.c:2920 kmem_cache_zalloc include/linux/slab.h:674 [inline] fill_pool+0x264/0x5c0 lib/debugobjects.c:171 __debug_object_init+0x7a/0xd10 lib/debugobjects.c:560 debug_hrtimer_init kernel/time/hrtimer.c:414 [inline] debug_init kernel/time/hrtimer.c:468 [inline] hrtimer_init+0x23/0x2b0 kernel/time/hrtimer.c:1444 tcp_init_xmit_timers+0x7b/0xc0 net/ipv4/tcp_timer.c:795 tcp_create_openreq_child+0x6b9/0x16c0 net/ipv4/tcp_minisocks.c:502 tcp_v4_syn_recv_sock+0x10c/0x1470 net/ipv4/tcp_ipv4.c:1541 tcp_v6_syn_recv_sock+0x157a/0x2350 net/ipv6/tcp_ipv6.c:1238 tcp_check_req+0x616/0x18c0 net/ipv4/tcp_minisocks.c:772 tcp_v4_rcv+0x221a/0x3780 net/ipv4/tcp_ipv4.c:2016 ip_protocol_deliver_rcu+0x5c/0x8a0 net/ipv4/ip_input.c:204 ip_local_deliver_finish+0x20a/0x370 net/ipv4/ip_input.c:231 NF_HOOK include/linux/netfilter.h:301 [inline] NF_HOOK include/linux/netfilter.h:295 [inline] ip_local_deliver+0x1b3/0x200 net/ipv4/ip_input.c:252 dst_input include/net/dst.h:458 [inline] ip_rcv_finish+0x1da/0x2f0 net/ipv4/ip_input.c:429 NF_HOOK include/linux/netfilter.h:301 [inline] NF_HOOK include/linux/netfilter.h:295 [inline] ip_rcv+0xaa/0xd0 net/ipv4/ip_input.c:540 __netif_receive_skb_one_core+0x114/0x180 net/core/dev.c:5436 __netif_receive_skb+0x27/0x1c0 net/core/dev.c:5550 process_backlog+0x232/0x6c0 net/core/dev.c:6413 __napi_poll+0xaf/0x440 net/core/dev.c:6961 napi_poll net/core/dev.c:7028 [inline] net_rx_action+0x801/0xb40 net/core/dev.c:7114 __do_softirq+0x29b/0x9f6 kernel/softirq.c:345 do_softirq.part.0+0xd9/0x130 kernel/softirq.c:248 do_softirq kernel/softirq.c:240 [inline] __local_bh_enable_ip+0x102/0x120 kernel/softirq.c:198 local_bh_enable include/linux/bottom_half.h:32 [inline] rcu_read_unlock_bh include/linux/rcupdate.h:745 [inline] ip_finish_output2+0x88b/0x21f0 net/ipv4/ip_output.c:231 __ip_finish_output net/ipv4/ip_output.c:308 [inline] __ip_finish_output+0x396/0x640 net/ipv4/ip_output.c:290 ip_finish_output+0x35/0x200 net/ipv4/ip_output.c:318 NF_HOOK_COND include/linux/netfilter.h:290 [inline] ip_output+0x196/0x310 net/ipv4/ip_output.c:432 dst_output include/net/dst.h:448 [inline] ip_local_out net/ipv4/ip_output.c:126 [inline] __ip_queue_xmit+0x905/0x1a20 net/ipv4/ip_output.c:533 __tcp_transmit_skb+0x1889/0x38f0 net/ipv4/tcp_output.c:1405 __tcp_send_ack.part.0+0x3d6/0x7a0 net/ipv4/tcp_output.c:3970 __tcp_send_ack net/ipv4/tcp_output.c:3976 [inline] tcp_send_ack+0x7d/0xa0 net/ipv4/tcp_output.c:3976 tcp_rcv_synsent_state_process net/ipv4/tcp_input.c:6186 [inline] tcp_rcv_state_process+0x3858/0x4bb0 net/ipv4/tcp_input.c:6355 tcp_v4_do_rcv+0x320/0x870 net/ipv4/tcp_ipv4.c:1716 sk_backlog_rcv include/net/sock.h:1016 [inline] __release_sock+0x134/0x3b0 net/core/sock.c:2556 release_sock+0x54/0x1b0 net/core/sock.c:3080 inet_stream_connect+0x76/0xa0 net/ipv4/af_inet.c:729 rds_tcp_conn_path_connect+0x61c/0x880 net/rds/tcp_connect.c:172 rds_connect_worker+0x1a5/0x2c0 net/rds/threads.c:176 process_one_work+0x98d/0x1600 kernel/workqueue.c:2275 worker_thread+0x64c/0x1120 kernel/workqueue.c:2421 kthread+0x3b1/0x4a0 kernel/kthread.c:292 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294