syzbot


INFO: rcu detected stall in ip_rcv (3)

Status: auto-closed as invalid on 2021/07/17 18:11
Subsystems: kernel
[Documentation on labels]
First crash: 1155d, last: 1155d
Similar bugs (9)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 BUG: soft lockup in ip_rcv C error 16 472d 1140d 0/1 upstream: reported C repro on 2021/05/04 03:32
upstream INFO: rcu detected stall in ip_rcv netfilter 2 1787d 1950d 0/27 auto-closed as invalid on 2019/10/25 14:11
upstream INFO: rcu detected stall in ip_rcv (2) net 1 1293d 1293d 0/27 auto-closed as invalid on 2021/03/01 14:28
linux-4.14 INFO: rcu detected stall in ip_rcv (2) 1 1304d 1304d 0/1 auto-closed as invalid on 2021/03/20 21:11
linux-4.14 INFO: rcu detected stall in ip_rcv 3 1484d 1503d 0/1 auto-closed as invalid on 2020/09/21 12:55
upstream INFO: rcu detected stall in ip_rcv (4) bpf net 12 782d 853d 0/27 auto-closed as invalid on 2022/07/25 09:18
upstream INFO: rcu detected stall in ip_rcv (5) net 2 297d 309d 0/27 auto-obsoleted due to no activity on 2023/11/23 01:36
linux-4.14 BUG: soft lockup in ip_rcv (2) 1 1176d 1176d 0/1 auto-closed as invalid on 2021/07/26 15:49
linux-4.14 BUG: soft lockup in ip_rcv (3) C error 4 652d 771d 0/1 upstream: reported C repro on 2022/05/07 10:08

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	0-....: (1 GPs behind) idle=176/1/0x4000000000000002 softirq=16935/16938 fqs=5236 
	(detected by 1, t=10502 jiffies, g=23477, q=226285)

============================================
WARNING: possible recursive locking detected
5.12.0-rc4-syzkaller #0 Not tainted
--------------------------------------------
kworker/u4:2/29 is trying to acquire lock:
ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_dump_cpu_stacks+0xaf/0x2a7 kernel/rcu/tree_stall.h:336

but task is already holding lock:
ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: print_other_cpu_stall kernel/rcu/tree_stall.h:541 [inline]
ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: check_cpu_stall kernel/rcu/tree_stall.h:706 [inline]
ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_pending kernel/rcu/tree.c:3830 [inline]
ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_sched_clock_irq.cold+0x2b9/0x11dd kernel/rcu/tree.c:2650

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(rcu_node_0);
  lock(rcu_node_0);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

9 locks held by kworker/u4:2/29:
 #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: atomic64_set include/asm-generic/atomic-instrumented.h:856 [inline]
 #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: atomic_long_set include/asm-generic/atomic-long.h:41 [inline]
 #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:616 [inline]
 #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline]
 #0: ffff88814790c138 ((wq_completion)krdsd){+.+.}-{0:0}, at: process_one_work+0x871/0x1600 kernel/workqueue.c:2246
 #1: ffffc90000e4fda8 ((work_completion)(&(&cp->cp_conn_w)->work)){+.+.}-{0:0}, at: process_one_work+0x8a5/0x1600 kernel/workqueue.c:2250
 #2: ffff888023ddc088 (&tc->t_conn_path_lock){+.+.}-{3:3}, at: rds_tcp_conn_path_connect+0x174/0x880 net/rds/tcp_connect.c:107
 #3: ffff8880142bb020 (k-sk_lock-AF_INET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1606 [inline]
 #3: ffff8880142bb020 (k-sk_lock-AF_INET){+.+.}-{0:0}, at: inet_stream_connect+0x3f/0xa0 net/ipv4/af_inet.c:727
 #4: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: read_pnet include/net/net_namespace.h:322 [inline]
 #4: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: sock_net include/net/sock.h:2560 [inline]
 #4: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __ip_queue_xmit+0x3f/0x1a20 net/ipv4/ip_output.c:458
 #5: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __skb_unlink include/linux/skbuff.h:2089 [inline]
 #5: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __skb_dequeue include/linux/skbuff.h:2104 [inline]
 #5: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: process_backlog+0x1c1/0x6c0 net/core/dev.c:6411
 #6: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: __skb_pull include/linux/skbuff.h:2320 [inline]
 #6: ffffffff8bf74360 (rcu_read_lock){....}-{1:2}, at: ip_local_deliver_finish+0x124/0x370 net/ipv4/ip_input.c:228
 #7: ffff8880144c3de0 (k-slock-AF_INET6){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
 #7: ffff8880144c3de0 (k-slock-AF_INET6){+.-.}-{2:2}, at: sk_clone_lock+0x296/0x1070 net/core/sock.c:1913
 #8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: print_other_cpu_stall kernel/rcu/tree_stall.h:541 [inline]
 #8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: check_cpu_stall kernel/rcu/tree_stall.h:706 [inline]
 #8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_pending kernel/rcu/tree.c:3830 [inline]
 #8: ffffffff8bf7c918 (rcu_node_0){-.-.}-{2:2}, at: rcu_sched_clock_irq.cold+0x2b9/0x11dd kernel/rcu/tree.c:2650

stack backtrace:
CPU: 1 PID: 29 Comm: kworker/u4:2 Not tainted 5.12.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: krdsd rds_connect_worker
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x141/0x1d7 lib/dump_stack.c:120
 print_deadlock_bug kernel/locking/lockdep.c:2829 [inline]
 check_deadlock kernel/locking/lockdep.c:2872 [inline]
 validate_chain kernel/locking/lockdep.c:3661 [inline]
 __lock_acquire.cold+0x14c/0x3b4 kernel/locking/lockdep.c:4900
 lock_acquire kernel/locking/lockdep.c:5510 [inline]
 lock_acquire+0x1ab/0x740 kernel/locking/lockdep.c:5475
 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
 _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
 rcu_dump_cpu_stacks+0xaf/0x2a7 kernel/rcu/tree_stall.h:336
 print_other_cpu_stall kernel/rcu/tree_stall.h:559 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:706 [inline]
 rcu_pending kernel/rcu/tree.c:3830 [inline]
 rcu_sched_clock_irq.cold+0x7e1/0x11dd kernel/rcu/tree.c:2650
 update_process_times+0x16d/0x200 kernel/time/timer.c:1796
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1369
 __run_hrtimer kernel/time/hrtimer.c:1537 [inline]
 __hrtimer_run_queues+0x1c0/0xe40 kernel/time/hrtimer.c:1601
 hrtimer_interrupt+0x330/0xa00 kernel/time/hrtimer.c:1663
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
 sysvec_apic_timer_interrupt+0x40/0xc0 arch/x86/kernel/apic/apic.c:1100
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:632
RIP: 0010:__orc_find+0x47/0xf0 arch/x86/kernel/unwind_orc.c:50
Code: 4c 39 e7 49 89 fd 0f 87 9a 00 00 00 49 bf 00 00 00 00 00 fc ff df 49 89 fe 48 89 fd eb 0c 48 8d 6b 04 49 89 de 49 39 ec 72 4d <4c> 89 e0 48 29 e8 48 89 c2 48 c1 e8 3f 48 c1 fa 02 48 01 d0 48 d1
RSP: 0018:ffffc90000dbffb0 EFLAGS: 00000246
RAX: ffffffff8783b9b5 RBX: ffffffff8e24280c RCX: ffffffff8783ba2c
RDX: 0000000000000000 RSI: ffffffff8eb85f7e RDI: ffffffff8e24280c
RBP: ffffffff8e242810 R08: 0000000000000000 R09: ffffffff8eb85f7e
R10: fffff520001b8025 R11: 0000000000084087 R12: ffffffff8e242810
R13: ffffffff8e24280c R14: ffffffff8e24280c R15: dffffc0000000000
 orc_find arch/x86/kernel/unwind_orc.c:173 [inline]
 unwind_next_frame+0x32a/0x1ce0 arch/x86/kernel/unwind_orc.c:443
 arch_stack_walk+0x7d/0xe0 arch/x86/kernel/stacktrace.c:25
 stack_trace_save+0x8c/0xc0 kernel/stacktrace.c:121
 kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38
 kasan_set_track mm/kasan/common.c:46 [inline]
 set_alloc_info mm/kasan/common.c:427 [inline]
 __kasan_slab_alloc+0x75/0x90 mm/kasan/common.c:460
 kasan_slab_alloc include/linux/kasan.h:223 [inline]
 slab_post_alloc_hook mm/slab.h:516 [inline]
 slab_alloc_node mm/slub.c:2907 [inline]
 slab_alloc mm/slub.c:2915 [inline]
 kmem_cache_alloc+0x155/0x370 mm/slub.c:2920
 kmem_cache_zalloc include/linux/slab.h:674 [inline]
 fill_pool+0x264/0x5c0 lib/debugobjects.c:171
 __debug_object_init+0x7a/0xd10 lib/debugobjects.c:560
 debug_hrtimer_init kernel/time/hrtimer.c:414 [inline]
 debug_init kernel/time/hrtimer.c:468 [inline]
 hrtimer_init+0x23/0x2b0 kernel/time/hrtimer.c:1444
 tcp_init_xmit_timers+0x7b/0xc0 net/ipv4/tcp_timer.c:795
 tcp_create_openreq_child+0x6b9/0x16c0 net/ipv4/tcp_minisocks.c:502
 tcp_v4_syn_recv_sock+0x10c/0x1470 net/ipv4/tcp_ipv4.c:1541
 tcp_v6_syn_recv_sock+0x157a/0x2350 net/ipv6/tcp_ipv6.c:1238
 tcp_check_req+0x616/0x18c0 net/ipv4/tcp_minisocks.c:772
 tcp_v4_rcv+0x221a/0x3780 net/ipv4/tcp_ipv4.c:2016
 ip_protocol_deliver_rcu+0x5c/0x8a0 net/ipv4/ip_input.c:204
 ip_local_deliver_finish+0x20a/0x370 net/ipv4/ip_input.c:231
 NF_HOOK include/linux/netfilter.h:301 [inline]
 NF_HOOK include/linux/netfilter.h:295 [inline]
 ip_local_deliver+0x1b3/0x200 net/ipv4/ip_input.c:252
 dst_input include/net/dst.h:458 [inline]
 ip_rcv_finish+0x1da/0x2f0 net/ipv4/ip_input.c:429
 NF_HOOK include/linux/netfilter.h:301 [inline]
 NF_HOOK include/linux/netfilter.h:295 [inline]
 ip_rcv+0xaa/0xd0 net/ipv4/ip_input.c:540
 __netif_receive_skb_one_core+0x114/0x180 net/core/dev.c:5436
 __netif_receive_skb+0x27/0x1c0 net/core/dev.c:5550
 process_backlog+0x232/0x6c0 net/core/dev.c:6413
 __napi_poll+0xaf/0x440 net/core/dev.c:6961
 napi_poll net/core/dev.c:7028 [inline]
 net_rx_action+0x801/0xb40 net/core/dev.c:7114
 __do_softirq+0x29b/0x9f6 kernel/softirq.c:345
 do_softirq.part.0+0xd9/0x130 kernel/softirq.c:248
 </IRQ>
 do_softirq kernel/softirq.c:240 [inline]
 __local_bh_enable_ip+0x102/0x120 kernel/softirq.c:198
 local_bh_enable include/linux/bottom_half.h:32 [inline]
 rcu_read_unlock_bh include/linux/rcupdate.h:745 [inline]
 ip_finish_output2+0x88b/0x21f0 net/ipv4/ip_output.c:231
 __ip_finish_output net/ipv4/ip_output.c:308 [inline]
 __ip_finish_output+0x396/0x640 net/ipv4/ip_output.c:290
 ip_finish_output+0x35/0x200 net/ipv4/ip_output.c:318
 NF_HOOK_COND include/linux/netfilter.h:290 [inline]
 ip_output+0x196/0x310 net/ipv4/ip_output.c:432
 dst_output include/net/dst.h:448 [inline]
 ip_local_out net/ipv4/ip_output.c:126 [inline]
 __ip_queue_xmit+0x905/0x1a20 net/ipv4/ip_output.c:533
 __tcp_transmit_skb+0x1889/0x38f0 net/ipv4/tcp_output.c:1405
 __tcp_send_ack.part.0+0x3d6/0x7a0 net/ipv4/tcp_output.c:3970
 __tcp_send_ack net/ipv4/tcp_output.c:3976 [inline]
 tcp_send_ack+0x7d/0xa0 net/ipv4/tcp_output.c:3976
 tcp_rcv_synsent_state_process net/ipv4/tcp_input.c:6186 [inline]
 tcp_rcv_state_process+0x3858/0x4bb0 net/ipv4/tcp_input.c:6355
 tcp_v4_do_rcv+0x320/0x870 net/ipv4/tcp_ipv4.c:1716
 sk_backlog_rcv include/net/sock.h:1016 [inline]
 __release_sock+0x134/0x3b0 net/core/sock.c:2556
 release_sock+0x54/0x1b0 net/core/sock.c:3080
 inet_stream_connect+0x76/0xa0 net/ipv4/af_inet.c:729
 rds_tcp_conn_path_connect+0x61c/0x880 net/rds/tcp_connect.c:172
 rds_connect_worker+0x1a5/0x2c0 net/rds/threads.c:176
 process_one_work+0x98d/0x1600 kernel/workqueue.c:2275
 worker_thread+0x64c/0x1120 kernel/workqueue.c:2421
 kthread+0x3b1/0x4a0 kernel/kthread.c:292
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/04/18 18:11 bpf-next cdf0e80e9fbe 7e2b734b .config console log report info ci-upstream-bpf-next-kasan-gce INFO: rcu detected stall in ip_rcv
* Struck through repros no longer work on HEAD.