bisecting cause commit starting from 7e062cda7d90543ac8c7700fc7c5527d0c0f22ad building syzkaller on 5783034f220fa03dd9407034a4804be5890c3377 testing commit 7e062cda7d90543ac8c7700fc7c5527d0c0f22ad compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c4361e91f20e72db35a2606b44841d5e5c7785a2fc6734ea8c0353e2b2e41072 all runs: crashed: WARNING: refcount bug in sk_psock_get testing release v5.18 testing commit 4b0986a3613c92f4ec1bdc7f60ec66fea135991f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 3f7ae539e661cdce76253bcb75d517ea2c7453789b4ba913090aeb27db23d214 all runs: crashed: WARNING: refcount bug in sk_psock_get testing release v5.17 testing commit f443e374ae131c168a065ea1748feac6b2e76613 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c5dcbc694b3f242d3a72f02a1dd0a7c2d24bef7177de6ff6e373f66d037ba354 all runs: crashed: WARNING: refcount bug in sk_psock_get testing release v5.16 testing commit df0cc57e057f18e44dac8e6c18aba47ab53202f9 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: eba2e8a76673ce2d00d4d0fccc6ecc953a0b75c628982b0703e8d2434e948cf0 all runs: OK # git bisect start f443e374ae131c168a065ea1748feac6b2e76613 df0cc57e057f18e44dac8e6c18aba47ab53202f9 Bisecting: 6995 revisions left to test after this (roughly 13 steps) [22ef12195e13c5ec58320dbf99ef85059a2c0820] Merge tag 'staging-5.17-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging testing commit 22ef12195e13c5ec58320dbf99ef85059a2c0820 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1c96585381948bc2517d283fce871810c98e0dabd158646895486b5ca3e917a6 all runs: OK # git bisect good 22ef12195e13c5ec58320dbf99ef85059a2c0820 Bisecting: 3520 revisions left to test after this (roughly 12 steps) [51620150ca2df62f8ea472ab8962be590c957288] cifs: update internal module number testing commit 51620150ca2df62f8ea472ab8962be590c957288 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 3869100e34d573df923060d4022c7e77d39f1099096af6a6bf6bd476418bf96b all runs: OK # git bisect good 51620150ca2df62f8ea472ab8962be590c957288 Bisecting: 1760 revisions left to test after this (roughly 11 steps) [d1ca60efc53d665cf89ed847a14a510a81770b81] netfilter: ctnetlink: disable helper autoassign testing commit d1ca60efc53d665cf89ed847a14a510a81770b81 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 17d1c974d9eb2135fafb8c51e0637549725a9aa29bcb5848a8923b73b16253fc all runs: crashed: WARNING: refcount bug in sk_psock_get # git bisect bad d1ca60efc53d665cf89ed847a14a510a81770b81 Bisecting: 879 revisions left to test after this (roughly 10 steps) [5475e8f03c80bbce7b43a57d861f5acc44a60b22] random: move the random sysctl declarations to its own file testing commit 5475e8f03c80bbce7b43a57d861f5acc44a60b22 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6a4851fd28fd435c8a5ce0700969bd7ba81ad98b34418526f8c04037c93bb65c all runs: OK # git bisect good 5475e8f03c80bbce7b43a57d861f5acc44a60b22 Bisecting: 439 revisions left to test after this (roughly 9 steps) [23a46422c56144939c091c76cf389aa863ce9c18] Merge tag 'net-5.17-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net testing commit 23a46422c56144939c091c76cf389aa863ce9c18 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: e64d54e855fd867643ffa4f5af78c7e9631fc1b0902dd7df7abb5e6b2f1b88da all runs: OK # git bisect good 23a46422c56144939c091c76cf389aa863ce9c18 Bisecting: 217 revisions left to test after this (roughly 8 steps) [bb37101b36332345a1e1c1f9f2f3bcc8ad7edb65] Merge tag 'tty-5.17-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty testing commit bb37101b36332345a1e1c1f9f2f3bcc8ad7edb65 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: ca79ef32929c24349c9f7aff4f13f39063da5c709318c1ffb4d1c2e79176b898 all runs: OK # git bisect good bb37101b36332345a1e1c1f9f2f3bcc8ad7edb65 Bisecting: 109 revisions left to test after this (roughly 7 steps) [c36c04c2e132fc39f6b658bf607aed4425427fd7] Revert "mm/gup: small refactoring: simplify try_grab_page()" testing commit c36c04c2e132fc39f6b658bf607aed4425427fd7 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 06f5c208ceab81f983e847e31c1ecd65fc129e548e76c74a52bfa43c98eedd7e all runs: OK # git bisect good c36c04c2e132fc39f6b658bf607aed4425427fd7 Bisecting: 54 revisions left to test after this (roughly 6 steps) [5b209d1a22afabfb7d644abb10510c5713a3e569] net/mlx5e: Avoid implicit modify hdr for decap drop rule testing commit 5b209d1a22afabfb7d644abb10510c5713a3e569 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5ed6d139ee085d5ceacdc06eb4cbb49defc2a6ad46a0a5de24e9cb5b74ac45a0 all runs: crashed: WARNING: refcount bug in sk_psock_get # git bisect bad 5b209d1a22afabfb7d644abb10510c5713a3e569 Bisecting: 27 revisions left to test after this (roughly 5 steps) [6533e558c6505e94c3e0ed4281ed5e31ec985f4d] i40e: Fix reset path while removing the driver testing commit 6533e558c6505e94c3e0ed4281ed5e31ec985f4d compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 7829890d3a6d8c38a108403fd22b133934f1375195fb6c1e5456b9b341d871d6 all runs: crashed: WARNING: refcount bug in sk_psock_get # git bisect bad 6533e558c6505e94c3e0ed4281ed5e31ec985f4d Bisecting: 13 revisions left to test after this (roughly 4 steps) [4e0f718daf97d47cf7dec122da1be970f145c809] ax25: improve the incomplete fix to avoid UAF and NPD bugs testing commit 4e0f718daf97d47cf7dec122da1be970f145c809 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: f0b0d9da8eccdfda9b13e514062c7266e67fa5cfa9482e2fc9870a23f8b7636c all runs: OK # git bisect good 4e0f718daf97d47cf7dec122da1be970f145c809 Bisecting: 7 revisions left to test after this (roughly 3 steps) [5d8a8b324ff48c9d9fe4f1634e33dc647d2481b4] MAINTAINERS: Remove Harry Morris bouncing address testing commit 5d8a8b324ff48c9d9fe4f1634e33dc647d2481b4 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: b4d1e58828acb64a961df1b63cc9d894182780853ccf1b1c0a582bb8e779d0ea run #0: basic kernel testing failed: BUG: program execution failed: executor NUM: failed to write control pipe: write |NUM: broken pipe run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 5d8a8b324ff48c9d9fe4f1634e33dc647d2481b4 Bisecting: 3 revisions left to test after this (roughly 2 steps) [010a2a6623317bbf13facaff8bf50ac08468c1df] Merge tag 'ieee802154-for-net-2022-01-28' of git://git.kernel.org/pub/scm/linux/kernel/git/sschmidt/wpan testing commit 010a2a6623317bbf13facaff8bf50ac08468c1df compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6719e9679b756d301e0a6ea9a68d577e6e581b0a46e540b7d57fef87f112a986 all runs: OK # git bisect good 010a2a6623317bbf13facaff8bf50ac08468c1df Bisecting: 1 revision left to test after this (roughly 1 step) [341adeec9adad0874f29a0a1af35638207352a39] net/smc: Forward wakeup to smc socket waitqueue after fallback testing commit 341adeec9adad0874f29a0a1af35638207352a39 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 21ce27fb193f985eee589734006f6ac1216483fa6bc9419c75950db8c6db43b0 all runs: crashed: WARNING: refcount bug in sk_psock_get # git bisect bad 341adeec9adad0874f29a0a1af35638207352a39 Bisecting: 0 revisions left to test after this (roughly 0 steps) [6449520391dfc3d2cef134f11a91251a054ff7d0] net: stmmac: properly handle with runtime pm in stmmac_dvr_remove() testing commit 6449520391dfc3d2cef134f11a91251a054ff7d0 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: bddfdbd6a02fe68b8b7c1e1bac0f7c286e155280ec0df7d879a041ea3d713c17 all runs: OK # git bisect good 6449520391dfc3d2cef134f11a91251a054ff7d0 341adeec9adad0874f29a0a1af35638207352a39 is the first bad commit commit 341adeec9adad0874f29a0a1af35638207352a39 Author: Wen Gu Date: Wed Jan 26 23:33:04 2022 +0800 net/smc: Forward wakeup to smc socket waitqueue after fallback When we replace TCP with SMC and a fallback occurs, there may be some socket waitqueue entries remaining in smc socket->wq, such as eppoll_entries inserted by userspace applications. After the fallback, data flows over TCP/IP and only clcsocket->wq will be woken up. Applications can't be notified by the entries which were inserted in smc socket->wq before fallback. So we need a mechanism to wake up smc socket->wq at the same time if some entries remaining in it. The current workaround is to transfer the entries from smc socket->wq to clcsock->wq during the fallback. But this may cause a crash like this: general protection fault, probably for non-canonical address 0xdead000000000100: 0000 [#1] PREEMPT SMP PTI CPU: 3 PID: 0 Comm: swapper/3 Kdump: loaded Tainted: G E 5.16.0+ #107 RIP: 0010:__wake_up_common+0x65/0x170 Call Trace: __wake_up_common_lock+0x7a/0xc0 sock_def_readable+0x3c/0x70 tcp_data_queue+0x4a7/0xc40 tcp_rcv_established+0x32f/0x660 ? sk_filter_trim_cap+0xcb/0x2e0 tcp_v4_do_rcv+0x10b/0x260 tcp_v4_rcv+0xd2a/0xde0 ip_protocol_deliver_rcu+0x3b/0x1d0 ip_local_deliver_finish+0x54/0x60 ip_local_deliver+0x6a/0x110 ? tcp_v4_early_demux+0xa2/0x140 ? tcp_v4_early_demux+0x10d/0x140 ip_sublist_rcv_finish+0x49/0x60 ip_sublist_rcv+0x19d/0x230 ip_list_rcv+0x13e/0x170 __netif_receive_skb_list_core+0x1c2/0x240 netif_receive_skb_list_internal+0x1e6/0x320 napi_complete_done+0x11d/0x190 mlx5e_napi_poll+0x163/0x6b0 [mlx5_core] __napi_poll+0x3c/0x1b0 net_rx_action+0x27c/0x300 __do_softirq+0x114/0x2d2 irq_exit_rcu+0xb4/0xe0 common_interrupt+0xba/0xe0 The crash is caused by privately transferring waitqueue entries from smc socket->wq to clcsock->wq. The owners of these entries, such as epoll, have no idea that the entries have been transferred to a different socket wait queue and still use original waitqueue spinlock (smc socket->wq.wait.lock) to make the entries operation exclusive, but it doesn't work. The operations to the entries, such as removing from the waitqueue (now is clcsock->wq after fallback), may cause a crash when clcsock waitqueue is being iterated over at the moment. This patch tries to fix this by no longer transferring wait queue entries privately, but introducing own implementations of clcsock's callback functions in fallback situation. The callback functions will forward the wakeup to smc socket->wq if clcsock->wq is actually woken up and smc socket->wq has remaining entries. Fixes: 2153bd1e3d3d ("net/smc: Transfer remaining wait queue entries during fallback") Suggested-by: Karsten Graul Signed-off-by: Wen Gu Acked-by: Karsten Graul Signed-off-by: David S. Miller net/smc/af_smc.c | 133 ++++++++++++++++++++++++++++++++++++++++++++++++------- net/smc/smc.h | 20 ++++++++- 2 files changed, 137 insertions(+), 16 deletions(-) culprit signature: 21ce27fb193f985eee589734006f6ac1216483fa6bc9419c75950db8c6db43b0 parent signature: bddfdbd6a02fe68b8b7c1e1bac0f7c286e155280ec0df7d879a041ea3d713c17 revisions tested: 18, total time: 3h54m22.057963521s (build: 1h55m6.434989638s, test: 1h57m30.885098203s) first bad commit: 341adeec9adad0874f29a0a1af35638207352a39 net/smc: Forward wakeup to smc socket waitqueue after fallback recipients (to): ["davem@davemloft.net" "guwen@linux.alibaba.com" "kgraul@linux.ibm.com"] recipients (cc): [] crash: WARNING: refcount bug in sk_psock_get ------------[ cut here ]------------ refcount_t: saturated; leaking memory. WARNING: CPU: 0 PID: 4078 at lib/refcount.c:19 refcount_warn_saturate+0x9d/0x140 lib/refcount.c:19 Modules linked in: CPU: 1 PID: 4078 Comm: syz-executor.0 Not tainted 5.17.0-rc1-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:refcount_warn_saturate+0x9d/0x140 lib/refcount.c:19 Code: 58 0f fe 08 01 e8 81 f6 af 04 0f 0b eb dd 80 3d 4c 0f fe 08 00 75 d4 48 c7 c7 40 53 42 89 c6 05 3c 0f fe 08 01 e8 61 f6 af 04 <0f> 0b eb bd 80 3d 2b 0f fe 08 00 75 b4 48 c7 c7 40 53 42 89 c6 05 RSP: 0018:ffffc900026ffa30 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff88807bf302b8 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000004 RDI: fffff520004dff38 RBP: 0000000000000000 R08: 0000000000000001 R09: ffff8880b9e27a2b R10: ffffed10173c4f45 R11: 0000000000000001 R12: 1ffff920004dff49 R13: ffff88807bf30000 R14: 0000000000000000 R15: ffff888079b76e20 FS: 00007f7b62997700(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fc6458d2300 CR3: 000000001deb9000 CR4: 00000000003506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __refcount_add_not_zero include/linux/refcount.h:163 [inline] __refcount_inc_not_zero include/linux/refcount.h:227 [inline] refcount_inc_not_zero include/linux/refcount.h:245 [inline] sk_psock_get+0x290/0x2c0 include/linux/skmsg.h:450 tls_data_ready+0x67/0x150 net/tls/tls_sw.c:2167 tcp_data_queue+0x1e7d/0x4c70 net/ipv4/tcp_input.c:5047 tcp_rcv_state_process+0xac9/0x4720 net/ipv4/tcp_input.c:6607 tcp_v4_do_rcv+0x299/0x7e0 net/ipv4/tcp_ipv4.c:1741 sk_backlog_rcv include/net/sock.h:1037 [inline] __release_sock+0x113/0x360 net/core/sock.c:2779 release_sock+0x4a/0x170 net/core/sock.c:3311 inet_shutdown+0x185/0x360 net/ipv4/af_inet.c:910 __sys_shutdown_sock net/socket.c:2252 [inline] __sys_shutdown_sock net/socket.c:2246 [inline] __sys_shutdown+0xcd/0x160 net/socket.c:2264 __do_sys_shutdown net/socket.c:2272 [inline] __se_sys_shutdown net/socket.c:2270 [inline] __x64_sys_shutdown+0x4b/0x70 net/socket.c:2270 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0x80 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7f7b63222109 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f7b62997168 EFLAGS: 00000246 ORIG_RAX: 0000000000000030 RAX: ffffffffffffffda RBX: 00007f7b63334f60 RCX: 00007f7b63222109 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000003 RBP: 00007f7b6327c08d R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007ffec2a27bff R14: 00007f7b62997300 R15: 0000000000022000