ci starts bisection 2023-06-10 17:46:59.036358975 +0000 UTC m=+289331.324785835 bisecting fixing commit since 0840a7914caa14315a3191178a9f72c742477860 building syzkaller on a371c43c33b6f901421f93b655442363c072d251 ensuring issue is reproducible on original commit 0840a7914caa14315a3191178a9f72c742477860 testing commit 0840a7914caa14315a3191178a9f72c742477860 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 0fd52fef7bc7c6b241852baeb06f21e58100224b101c082152b4de3b5f735d3f run #0: crashed: possible deadlock in rds_wake_sk_sleep run #1: crashed: possible deadlock in rds_wake_sk_sleep run #2: crashed: possible deadlock in rds_wake_sk_sleep run #3: crashed: possible deadlock in rds_wake_sk_sleep run #4: crashed: possible deadlock in rds_wake_sk_sleep run #5: crashed: possible deadlock in rds_wake_sk_sleep run #6: crashed: possible deadlock in rds_wake_sk_sleep run #7: crashed: possible deadlock in rds_wake_sk_sleep run #8: crashed: possible deadlock in rds_wake_sk_sleep run #9: crashed: possible deadlock in rds_wake_sk_sleep run #10: crashed: possible deadlock in rds_wake_sk_sleep run #11: crashed: possible deadlock in rds_wake_sk_sleep run #12: crashed: possible deadlock in rds_wake_sk_sleep run #13: crashed: possible deadlock in rds_wake_sk_sleep run #14: crashed: possible deadlock in rds_wake_sk_sleep run #15: crashed: possible deadlock in rds_wake_sk_sleep run #16: crashed: possible deadlock in rds_wake_sk_sleep run #17: crashed: possible deadlock in rds_wake_sk_sleep run #18: OK run #19: OK testing current HEAD 64569520920a3ca5d456ddd9f4f95fc6ea9b8b45 testing commit 64569520920a3ca5d456ddd9f4f95fc6ea9b8b45 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 3c10df01a5d04c18fb9165e33ff0cd9bfdfd750acf64c4b67a3ae83d61bdfb52 run #0: crashed: possible deadlock in rds_wake_sk_sleep run #1: crashed: possible deadlock in rds_wake_sk_sleep run #2: crashed: possible deadlock in rds_wake_sk_sleep run #3: crashed: possible deadlock in rds_wake_sk_sleep run #4: crashed: possible deadlock in rds_wake_sk_sleep run #5: crashed: possible deadlock in rds_wake_sk_sleep run #6: crashed: possible deadlock in rds_wake_sk_sleep run #7: crashed: possible deadlock in rds_wake_sk_sleep run #8: crashed: possible deadlock in rds_wake_sk_sleep run #9: OK crash still not fixed/happens on the oldest tested release revisions tested: 2, total time: 48m49.765766475s (build: 24m27.051615071s, test: 23m3.212189175s) crash still not fixed on HEAD or HEAD had kernel test errors commit msg: Merge tag 'block-6.4-2023-06-09' of git://git.kernel.dk/linux crash: possible deadlock in rds_wake_sk_sleep ====================================================== WARNING: possible circular locking dependency detected 6.4.0-rc5-syzkaller #0 Not tainted ------------------------------------------------------ kworker/u4:5/1011 is trying to acquire lock: ffff88802a28b630 (&rs->rs_recv_lock){...-}-{2:2}, at: rds_wake_sk_sleep+0x1a/0xc0 net/rds/af_rds.c:109 but task is already holding lock: ffff888020ffd100 (&rm->m_rs_lock){..-.}-{2:2}, at: rds_send_remove_from_sock+0x1e7/0x9a0 net/rds/send.c:628 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #1 (&rm->m_rs_lock){..-.}-{2:2}: __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline] _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:162 rds_message_purge net/rds/message.c:138 [inline] rds_message_put+0x16d/0xab0 net/rds/message.c:180 rds_clear_recv_queue+0x1c5/0x350 net/rds/recv.c:768 rds_release+0xca/0x350 net/rds/af_rds.c:73 __sock_release+0xbb/0x280 net/socket.c:653 sock_close+0xf/0x20 net/socket.c:1397 __fput+0x1fa/0x9a0 fs/file_table.c:321 task_work_run+0x12b/0x220 kernel/task_work.c:179 resume_user_mode_work include/linux/resume_user_mode.h:49 [inline] exit_to_user_mode_loop kernel/entry/common.c:171 [inline] exit_to_user_mode_prepare+0x210/0x240 kernel/entry/common.c:204 __syscall_exit_to_user_mode_work kernel/entry/common.c:286 [inline] syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:297 do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 entry_SYSCALL_64_after_hwframe+0x63/0xcd -> #0 (&rs->rs_recv_lock){...-}-{2:2}: check_prev_add kernel/locking/lockdep.c:3113 [inline] check_prevs_add kernel/locking/lockdep.c:3232 [inline] validate_chain kernel/locking/lockdep.c:3847 [inline] __lock_acquire+0x2fcd/0x5f30 kernel/locking/lockdep.c:5088 lock_acquire kernel/locking/lockdep.c:5705 [inline] lock_acquire+0x1ad/0x520 kernel/locking/lockdep.c:5670 __raw_read_lock_irqsave include/linux/rwlock_api_smp.h:160 [inline] _raw_read_lock_irqsave+0x45/0x90 kernel/locking/spinlock.c:236 rds_wake_sk_sleep+0x1a/0xc0 net/rds/af_rds.c:109 rds_send_remove_from_sock+0x256/0x9a0 net/rds/send.c:634 rds_send_path_drop_acked+0x276/0x360 net/rds/send.c:710 rds_tcp_write_space+0x196/0x5c0 net/rds/tcp_send.c:198 tcp_new_space net/ipv4/tcp_input.c:5483 [inline] tcp_check_space+0xde/0x730 net/ipv4/tcp_input.c:5502 tcp_data_snd_check net/ipv4/tcp_input.c:5511 [inline] tcp_rcv_established+0x75f/0x2040 net/ipv4/tcp_input.c:6021 tcp_v4_do_rcv+0x52e/0x7f0 net/ipv4/tcp_ipv4.c:1722 sk_backlog_rcv include/net/sock.h:1115 [inline] __release_sock+0x113/0x360 net/core/sock.c:2917 release_sock+0x4a/0x170 net/core/sock.c:3484 rds_send_xmit+0x87e/0x2370 net/rds/send.c:422 rds_send_worker+0x77/0x270 net/rds/threads.c:200 process_one_work+0x86e/0x1410 kernel/workqueue.c:2405 worker_thread+0x5ab/0xef0 kernel/workqueue.c:2552 kthread+0x2e6/0x3c0 kernel/kthread.c:379 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308 other info that might help us debug this: Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&rm->m_rs_lock); lock(&rs->rs_recv_lock); lock(&rm->m_rs_lock); rlock(&rs->rs_recv_lock); *** DEADLOCK *** 5 locks held by kworker/u4:5/1011: #0: ffff88814a21a138 ((wq_completion)krdsd){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: ffff88814a21a138 ((wq_completion)krdsd){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline] #0: ffff88814a21a138 ((wq_completion)krdsd){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1324 [inline] #0: ffff88814a21a138 ((wq_completion)krdsd){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:643 [inline] #0: ffff88814a21a138 ((wq_completion)krdsd){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:670 [inline] #0: ffff88814a21a138 ((wq_completion)krdsd){+.+.}-{0:0}, at: process_one_work+0x793/0x1410 kernel/workqueue.c:2376 #1: ffffc90004db7dc0 ((work_completion)(&(&cp->cp_send_w)->work)){+.+.}-{0:0}, at: process_one_work+0x7c0/0x1410 kernel/workqueue.c:2380 #2: ffff888076192db0 (k-sk_lock-AF_INET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1709 [inline] #2: ffff888076192db0 (k-sk_lock-AF_INET){+.+.}-{0:0}, at: tcp_sock_set_cork+0xe/0x70 net/ipv4/tcp.c:3342 #3: ffff888076193038 (k-clock-AF_INET){++.-}-{2:2}, at: rds_tcp_write_space+0x20/0x5c0 net/rds/tcp_send.c:184 #4: ffff888020ffd100 (&rm->m_rs_lock){..-.}-{2:2}, at: rds_send_remove_from_sock+0x1e7/0x9a0 net/rds/send.c:628 stack backtrace: CPU: 1 PID: 1011 Comm: kworker/u4:5 Not tainted 6.4.0-rc5-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/25/2023 Workqueue: krdsd rds_send_worker Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x60/0xa0 lib/dump_stack.c:106 check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2188 check_prev_add kernel/locking/lockdep.c:3113 [inline] check_prevs_add kernel/locking/lockdep.c:3232 [inline] validate_chain kernel/locking/lockdep.c:3847 [inline] __lock_acquire+0x2fcd/0x5f30 kernel/locking/lockdep.c:5088 lock_acquire kernel/locking/lockdep.c:5705 [inline] lock_acquire+0x1ad/0x520 kernel/locking/lockdep.c:5670 __raw_read_lock_irqsave include/linux/rwlock_api_smp.h:160 [inline] _raw_read_lock_irqsave+0x45/0x90 kernel/locking/spinlock.c:236 rds_wake_sk_sleep+0x1a/0xc0 net/rds/af_rds.c:109 rds_send_remove_from_sock+0x256/0x9a0 net/rds/send.c:634 rds_send_path_drop_acked+0x276/0x360 net/rds/send.c:710 rds_tcp_write_space+0x196/0x5c0 net/rds/tcp_send.c:198 tcp_new_space net/ipv4/tcp_input.c:5483 [inline] tcp_check_space+0xde/0x730 net/ipv4/tcp_input.c:5502 tcp_data_snd_check net/ipv4/tcp_input.c:5511 [inline] tcp_rcv_established+0x75f/0x2040 net/ipv4/tcp_input.c:6021 tcp_v4_do_rcv+0x52e/0x7f0 net/ipv4/tcp_ipv4.c:1722 sk_backlog_rcv include/net/sock.h:1115 [inline] __release_sock+0x113/0x360 net/core/sock.c:2917 release_sock+0x4a/0x170 net/core/sock.c:3484 rds_send_xmit+0x87e/0x2370 net/rds/send.c:422 rds_send_worker+0x77/0x270 net/rds/threads.c:200 process_one_work+0x86e/0x1410 kernel/workqueue.c:2405 worker_thread+0x5ab/0xef0 kernel/workqueue.c:2552 kthread+0x2e6/0x3c0 kernel/kthread.c:379 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308