======================================================
WARNING: possible circular locking dependency detected
6.8.0-rc3-syzkaller-g99bd3cb0d12e #0 Not tainted
------------------------------------------------------
kworker/u4:1/12 is trying to acquire lock:
ffff88801e510620 (&rs->rs_recv_lock){...-}-{2:2}, at: rds_wake_sk_sleep+0x2e/0xd0 net/rds/af_rds.c:109
but task is already holding lock:
ffff88801eaaf100 (&rm->m_rs_lock){..-.}-{2:2}, at: rds_send_remove_from_sock+0x129/0x800 net/rds/send.c:628
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&rm->m_rs_lock){..-.}-{2:2}:
lock_acquire+0x1e3/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
rds_message_purge net/rds/message.c:138 [inline]
rds_message_put+0x14a/0xab0 net/rds/message.c:180
rds_inc_put net/rds/recv.c:83 [inline]
rds_clear_recv_queue+0x2e8/0x3c0 net/rds/recv.c:768
rds_release+0xc5/0x2e0 net/rds/af_rds.c:73
__sock_release net/socket.c:659 [inline]
sock_close+0xbe/0x240 net/socket.c:1421
__fput+0x42b/0x8a0 fs/file_table.c:376
__do_sys_close fs/open.c:1554 [inline]
__se_sys_close fs/open.c:1539 [inline]
__x64_sys_close+0x7e/0x110 fs/open.c:1539
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6f/0x77
-> #0 (&rs->rs_recv_lock){...-}-{2:2}:
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18ca/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1345/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e3/0x530 kernel/locking/lockdep.c:5754
__raw_read_lock_irqsave include/linux/rwlock_api_smp.h:160 [inline]
_raw_read_lock_irqsave+0xdd/0x130 kernel/locking/spinlock.c:236
rds_wake_sk_sleep+0x2e/0xd0 net/rds/af_rds.c:109
rds_send_remove_from_sock+0x1c9/0x800 net/rds/send.c:634
rds_send_path_drop_acked+0x36b/0x3b0 net/rds/send.c:710
rds_tcp_write_space+0x196/0x5f0 net/rds/tcp_send.c:199
tcp_new_space net/ipv4/tcp_input.c:5626 [inline]
tcp_check_space+0x660/0xb00 net/ipv4/tcp_input.c:5645
tcp_data_snd_check net/ipv4/tcp_input.c:5654 [inline]
tcp_rcv_established+0x101c/0x2030 net/ipv4/tcp_input.c:6172
tcp_v4_do_rcv+0x8dd/0xb90 net/ipv4/tcp_ipv4.c:1906
tcp_v4_rcv+0x2cba/0x35c0 net/ipv4/tcp_ipv4.c:2329
ip_protocol_deliver_rcu+0x225/0x430 net/ipv4/ip_input.c:205
ip_local_deliver_finish+0x2b6/0x500 net/ipv4/ip_input.c:233
NF_HOOK+0x3a6/0x450 include/linux/netfilter.h:314
NF_HOOK+0x3a6/0x450 include/linux/netfilter.h:314
__netif_receive_skb_one_core net/core/dev.c:5534 [inline]
__netif_receive_skb+0x2bf/0x6a0 net/core/dev.c:5648
process_backlog+0x385/0x760 net/core/dev.c:5976
__napi_poll+0xcd/0x490 net/core/dev.c:6576
napi_poll net/core/dev.c:6645 [inline]
net_rx_action+0x78b/0x1010 net/core/dev.c:6778
__do_softirq+0x2bd/0x942 kernel/softirq.c:553
do_softirq+0x11b/0x1e0 kernel/softirq.c:454
__local_bh_enable_ip+0x1bb/0x200 kernel/softirq.c:381
rds_tcp_tune+0x46b/0x630
rds_tcp_accept_one+0x32f/0xbb0 net/rds/tcp_listen.c:137
rds_tcp_accept_worker+0x3f/0xa0 net/rds/tcp.c:532
process_one_work kernel/workqueue.c:2633 [inline]
process_scheduled_works+0x915/0x1420 kernel/workqueue.c:2706
worker_thread+0xa5f/0x1000 kernel/workqueue.c:2787
kthread+0x2f1/0x390 kernel/kthread.c:388
ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1b/0x30 arch/x86/entry/entry_64.S:242
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&rm->m_rs_lock);
lock(&rs->rs_recv_lock);
lock(&rm->m_rs_lock);
rlock(&rs->rs_recv_lock);
*** DEADLOCK ***
7 locks held by kworker/u4:1/12:
#0: ffff88802ae91d38 ((wq_completion)krdsd){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:2608 [inline]
#0: ffff88802ae91d38 ((wq_completion)krdsd){+.+.}-{0:0}, at: process_scheduled_works+0x825/0x1420 kernel/workqueue.c:2706
#1: ffffc90000117d20 ((work_completion)(&rtn->rds_tcp_accept_w)){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:2608 [inline]
#1: ffffc90000117d20 ((work_completion)(&rtn->rds_tcp_accept_w)){+.+.}-{0:0}, at: process_scheduled_works+0x825/0x1420 kernel/workqueue.c:2706
#2: ffffffff8e130ae0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
#2: ffffffff8e130ae0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
#2: ffffffff8e130ae0 (rcu_read_lock){....}-{1:2}, at: process_backlog+0x2cd/0x760 net/core/dev.c:5975
#3: ffffffff8e130ae0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
#3: ffffffff8e130ae0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
#3: ffffffff8e130ae0 (rcu_read_lock){....}-{1:2}, at: ip_local_deliver_finish+0x1bc/0x500 net/ipv4/ip_input.c:232
#4: ffff88801e3b5670 (k-slock-AF_INET/1){+.-.}-{2:2}, at: tcp_v4_rcv+0x2c69/0x35c0 net/ipv4/tcp_ipv4.c:2325
#5: ffff88801e3b5978 (k-clock-AF_INET){++.-}-{2:2}, at: rds_tcp_write_space+0x33/0x5f0 net/rds/tcp_send.c:185
#6: ffff88801eaaf100 (&rm->m_rs_lock){..-.}-{2:2}, at: rds_send_remove_from_sock+0x129/0x800 net/rds/send.c:628
stack backtrace:
CPU: 0 PID: 12 Comm: kworker/u4:1 Not tainted 6.8.0-rc3-syzkaller-g99bd3cb0d12e #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/25/2024
Workqueue: krdsd rds_tcp_accept_worker
Call Trace:
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2e0 lib/dump_stack.c:106
check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18ca/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1345/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e3/0x530 kernel/locking/lockdep.c:5754
__raw_read_lock_irqsave include/linux/rwlock_api_smp.h:160 [inline]
_raw_read_lock_irqsave+0xdd/0x130 kernel/locking/spinlock.c:236
rds_wake_sk_sleep+0x2e/0xd0 net/rds/af_rds.c:109
rds_send_remove_from_sock+0x1c9/0x800 net/rds/send.c:634
rds_send_path_drop_acked+0x36b/0x3b0 net/rds/send.c:710
rds_tcp_write_space+0x196/0x5f0 net/rds/tcp_send.c:199
tcp_new_space net/ipv4/tcp_input.c:5626 [inline]
tcp_check_space+0x660/0xb00 net/ipv4/tcp_input.c:5645
tcp_data_snd_check net/ipv4/tcp_input.c:5654 [inline]
tcp_rcv_established+0x101c/0x2030 net/ipv4/tcp_input.c:6172
tcp_v4_do_rcv+0x8dd/0xb90 net/ipv4/tcp_ipv4.c:1906
tcp_v4_rcv+0x2cba/0x35c0 net/ipv4/tcp_ipv4.c:2329
ip_protocol_deliver_rcu+0x225/0x430 net/ipv4/ip_input.c:205
ip_local_deliver_finish+0x2b6/0x500 net/ipv4/ip_input.c:233
NF_HOOK+0x3a6/0x450 include/linux/netfilter.h:314
NF_HOOK+0x3a6/0x450 include/linux/netfilter.h:314
__netif_receive_skb_one_core net/core/dev.c:5534 [inline]
__netif_receive_skb+0x2bf/0x6a0 net/core/dev.c:5648
process_backlog+0x385/0x760 net/core/dev.c:5976
__napi_poll+0xcd/0x490 net/core/dev.c:6576
napi_poll net/core/dev.c:6645 [inline]
net_rx_action+0x78b/0x1010 net/core/dev.c:6778
__do_softirq+0x2bd/0x942 kernel/softirq.c:553
do_softirq+0x11b/0x1e0 kernel/softirq.c:454
__local_bh_enable_ip+0x1bb/0x200 kernel/softirq.c:381
rds_tcp_tune+0x46b/0x630
rds_tcp_accept_one+0x32f/0xbb0 net/rds/tcp_listen.c:137
rds_tcp_accept_worker+0x3f/0xa0 net/rds/tcp.c:532
process_one_work kernel/workqueue.c:2633 [inline]
process_scheduled_works+0x915/0x1420 kernel/workqueue.c:2706
worker_thread+0xa5f/0x1000 kernel/workqueue.c:2787
kthread+0x2f1/0x390 kernel/kthread.c:388
ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1b/0x30 arch/x86/entry/entry_64.S:242