ci starts bisection 2022-11-24 16:26:10.345145955 +0000 UTC m=+1285.728051285 bisecting cause commit starting from 4312098baf37ee17a8350725e6e0d0e8590252d4 building syzkaller on 12c66417513689207e96cb377eba36af9bf78535 ensuring issue is reproducible on original commit 4312098baf37ee17a8350725e6e0d0e8590252d4 testing commit 4312098baf37ee17a8350725e6e0d0e8590252d4 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: f8b4ff9140839bb8a2826335c8953b14e486787d0a7791d5a2fc6f89e6f66e3f all runs: crashed: WARNING in inet_csk_destroy_sock testing release v6.0 testing commit 4fe89d07dcc2804c8b562f6c7896a45643d34b2f gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 58f4c14437b3b62da07ce65a6ae16afd90d49497fffe3d1910918abc2f591ead all runs: OK # git bisect start 4312098baf37ee17a8350725e6e0d0e8590252d4 4fe89d07dcc2804c8b562f6c7896a45643d34b2f Bisecting: 7145 revisions left to test after this (roughly 13 steps) [e08466a7c00733a501d3c5328d29ec974478d717] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma testing commit e08466a7c00733a501d3c5328d29ec974478d717 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 80ff24105e6c546e602a5df2875359e4c7fe3876c8ec53e19666509a449a99ff all runs: crashed: WARNING in inet_csk_destroy_sock # git bisect bad e08466a7c00733a501d3c5328d29ec974478d717 Bisecting: 3980 revisions left to test after this (roughly 12 steps) [a47e60729d9624e931f988709ab76e043e2ee8b9] Merge tag 'backlight-next-6.1' of git://git.kernel.org/pub/scm/linux/kernel/git/lee/backlight testing commit a47e60729d9624e931f988709ab76e043e2ee8b9 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: b43b417909f14d96242a4c946086c5e58a6073244152e08299ad9fdb8f81e841 all runs: crashed: WARNING in inet_csk_destroy_sock # git bisect bad a47e60729d9624e931f988709ab76e043e2ee8b9 Bisecting: 1573 revisions left to test after this (roughly 11 steps) [915b96c52763e2988e6368b538b487a7138b8fa4] Merge tag 'wireless-next-2022-09-30' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next testing commit 915b96c52763e2988e6368b538b487a7138b8fa4 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 79e9cbfccd8792c89c7fa94ca1d608f48dd434e8b00878866c125bd0e9cb9f63 all runs: crashed: WARNING in inet_csk_destroy_sock # git bisect bad 915b96c52763e2988e6368b538b487a7138b8fa4 Bisecting: 837 revisions left to test after this (roughly 10 steps) [2c119d9982b1aba54a2eca59c2455cd09f3bc749] net: dsa: microchip: add the support for set_ageing_time testing commit 2c119d9982b1aba54a2eca59c2455cd09f3bc749 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 0934622f7d8ebae7ca13b6bfd75d8ac2fe588e7a3eb0823b24c0d77e796f67d6 all runs: crashed: WARNING in inet_csk_destroy_sock # git bisect bad 2c119d9982b1aba54a2eca59c2455cd09f3bc749 Bisecting: 413 revisions left to test after this (roughly 9 steps) [744ccd5c64bda0a4130b470ce2772985f20913ce] Merge branch 'net-sched-remove-unused-variables' testing commit 744ccd5c64bda0a4130b470ce2772985f20913ce gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5f6e7b5743f2e6efd3aeed5768eec1914d9b0a00ac9fd51f395e7fbf63149077 all runs: crashed: WARNING in inet_csk_destroy_sock # git bisect bad 744ccd5c64bda0a4130b470ce2772985f20913ce Bisecting: 206 revisions left to test after this (roughly 8 steps) [73ef239cd8439b33273cd95d08cffaf8022b01a8] net: marvell: prestera: implement br_port_locked flag offloading testing commit 73ef239cd8439b33273cd95d08cffaf8022b01a8 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 39a3a4018256c4a1b05d5a339e71c8158b0c6d62fd5a60d03422f4e73f9efa10 all runs: OK # git bisect good 73ef239cd8439b33273cd95d08cffaf8022b01a8 Bisecting: 89 revisions left to test after this (roughly 7 steps) [643952f3ecace9e20b8a0c5cd1bbd7409ac2d02c] Merge tag 'wireless-next-2022-08-26-v2' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next testing commit 643952f3ecace9e20b8a0c5cd1bbd7409ac2d02c gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: bceb89ed8f1d2ed401a98f132eb22905c70541269975e445443ae10dd97317fa all runs: crashed: WARNING in inet_csk_destroy_sock # git bisect bad 643952f3ecace9e20b8a0c5cd1bbd7409ac2d02c Bisecting: 56 revisions left to test after this (roughly 6 steps) [77baa37a9be948562c8b7630e3afd9d497e14550] Merge branch '100GbE' of git://git.kernel.org/pub/scm/linux/kernel/git/tnguy/next-queue testing commit 77baa37a9be948562c8b7630e3afd9d497e14550 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: ac72529f57dbedb7ec54a8a52af19e8048355e08a0d9249a370478610fc7f5f2 all runs: crashed: WARNING in inet_csk_destroy_sock # git bisect bad 77baa37a9be948562c8b7630e3afd9d497e14550 Bisecting: 29 revisions left to test after this (roughly 5 steps) [1be9ac87a75a4fc0e2cc254e412d2d67a58a7191] selftests/net: Add sk_bind_sendto_listen and sk_connect_zero_addr testing commit 1be9ac87a75a4fc0e2cc254e412d2d67a58a7191 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c4d471bf005d196507105fcfbd6ffe22a14c752c0fb70ad5b5f40af11075f7a0 all runs: crashed: WARNING in inet_csk_destroy_sock # git bisect bad 1be9ac87a75a4fc0e2cc254e412d2d67a58a7191 Bisecting: 14 revisions left to test after this (roughly 4 steps) [72e0bcd1563602168391ea52157bdd82e6d7875a] net/mlx5: TC, Add support for SF tunnel offload testing commit 72e0bcd1563602168391ea52157bdd82e6d7875a gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 213f4b9bc8b23d02e4c7afa6d35d154ea354beb0e4aab1befe396293de73ba22 all runs: OK # git bisect good 72e0bcd1563602168391ea52157bdd82e6d7875a Bisecting: 7 revisions left to test after this (roughly 3 steps) [133706a960de413cea12f4f6e1a2262adb381f62] r8169: remove support for chip version 50 testing commit 133706a960de413cea12f4f6e1a2262adb381f62 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: cba72bfeddb2293b17ed51bcaa1245f8f713966d257a62e4a3bbde7079e39308 all runs: OK # git bisect good 133706a960de413cea12f4f6e1a2262adb381f62 Bisecting: 3 revisions left to test after this (roughly 2 steps) [35bbe652c421037822aba29423f5f1f7d0d69f3f] net: ethernet: ti: davinci_mdio: fix build for mdio bitbang uses testing commit 35bbe652c421037822aba29423f5f1f7d0d69f3f gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a9d5dc0aba89293e168fc3063dee352f38855fae83b1be0e0796f7f7fccb6fa3 all runs: OK # git bisect good 35bbe652c421037822aba29423f5f1f7d0d69f3f Bisecting: 1 revision left to test after this (roughly 1 step) [28044fc1d4953b07acec0da4d2fc4784c57ea6fb] net: Add a bhash2 table hashed by port and address testing commit 28044fc1d4953b07acec0da4d2fc4784c57ea6fb gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 9344a9518cba94e6b3aca9ad14aaed3a329ee7a9359774faf0a609d00b2ff40d all runs: crashed: WARNING in inet_csk_destroy_sock # git bisect bad 28044fc1d4953b07acec0da4d2fc4784c57ea6fb Bisecting: 0 revisions left to test after this (roughly 0 steps) [0bf73255d3a3cf3b0416e95f2c9f7c53095c2e1a] netlink: fix some kernel-doc comments testing commit 0bf73255d3a3cf3b0416e95f2c9f7c53095c2e1a gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5a754e19d0bd02f7a6262448050aadbafe040933ad1f574f379bf1502ce79e11 all runs: OK # git bisect good 0bf73255d3a3cf3b0416e95f2c9f7c53095c2e1a 28044fc1d4953b07acec0da4d2fc4784c57ea6fb is the first bad commit commit 28044fc1d4953b07acec0da4d2fc4784c57ea6fb Author: Joanne Koong Date: Mon Aug 22 11:10:21 2022 -0700 net: Add a bhash2 table hashed by port and address The current bind hashtable (bhash) is hashed by port only. In the socket bind path, we have to check for bind conflicts by traversing the specified port's inet_bind_bucket while holding the hashbucket's spinlock (see inet_csk_get_port() and inet_csk_bind_conflict()). In instances where there are tons of sockets hashed to the same port at different addresses, the bind conflict check is time-intensive and can cause softirq cpu lockups, as well as stops new tcp connections since __inet_inherit_port() also contests for the spinlock. This patch adds a second bind table, bhash2, that hashes by port and sk->sk_rcv_saddr (ipv4) and sk->sk_v6_rcv_saddr (ipv6). Searching the bhash2 table leads to significantly faster conflict resolution and less time holding the hashbucket spinlock. Please note a few things: * There can be the case where the a socket's address changes after it has been bound. There are two cases where this happens: 1) The case where there is a bind() call on INADDR_ANY (ipv4) or IPV6_ADDR_ANY (ipv6) and then a connect() call. The kernel will assign the socket an address when it handles the connect() 2) In inet_sk_reselect_saddr(), which is called when rebuilding the sk header and a few pre-conditions are met (eg rerouting fails). In these two cases, we need to update the bhash2 table by removing the entry for the old address, and add a new entry reflecting the updated address. * The bhash2 table must have its own lock, even though concurrent accesses on the same port are protected by the bhash lock. Bhash2 must have its own lock to protect against cases where sockets on different ports hash to different bhash hashbuckets but to the same bhash2 hashbucket. This brings up a few stipulations: 1) When acquiring both the bhash and the bhash2 lock, the bhash2 lock will always be acquired after the bhash lock and released before the bhash lock is released. 2) There are no nested bhash2 hashbucket locks. A bhash2 lock is always acquired+released before another bhash2 lock is acquired+released. * The bhash table cannot be superseded by the bhash2 table because for bind requests on INADDR_ANY (ipv4) or IPV6_ADDR_ANY (ipv6), every socket bound to that port must be checked for a potential conflict. The bhash table is the only source of port->socket associations. Signed-off-by: Joanne Koong Signed-off-by: Jakub Kicinski include/net/inet_connection_sock.h | 3 + include/net/inet_hashtables.h | 80 ++++++++++- include/net/sock.h | 14 ++ net/dccp/ipv4.c | 25 +++- net/dccp/ipv6.c | 18 +++ net/dccp/proto.c | 34 ++++- net/ipv4/af_inet.c | 26 +++- net/ipv4/inet_connection_sock.c | 275 ++++++++++++++++++++++++++++--------- net/ipv4/inet_hashtables.c | 268 ++++++++++++++++++++++++++++++++++-- net/ipv4/tcp.c | 11 +- net/ipv4/tcp_ipv4.c | 23 +++- net/ipv6/tcp_ipv6.c | 17 +++ 12 files changed, 700 insertions(+), 94 deletions(-) culprit signature: 9344a9518cba94e6b3aca9ad14aaed3a329ee7a9359774faf0a609d00b2ff40d parent signature: 5a754e19d0bd02f7a6262448050aadbafe040933ad1f574f379bf1502ce79e11 revisions tested: 16, total time: 4h15m27.46403893s (build: 2h22m11.544866679s, test: 1h50m37.992064712s) first bad commit: 28044fc1d4953b07acec0da4d2fc4784c57ea6fb net: Add a bhash2 table hashed by port and address recipients (to): ["davem@davemloft.net" "dccp@vger.kernel.org" "dsahern@kernel.org" "edumazet@google.com" "joannelkoong@gmail.com" "kuba@kernel.org" "kuba@kernel.org" "netdev@vger.kernel.org" "pabeni@redhat.com" "yoshfuji@linux-ipv6.org"] recipients (cc): ["gnault@redhat.com" "hbh25y@gmail.com" "joannelkoong@gmail.com" "kafai@fb.com" "kuniyu@amazon.co.jp" "linux-kernel@vger.kernel.org" "socketcan@hartkopp.net"] crash: WARNING in inet_csk_destroy_sock ------------[ cut here ]------------ WARNING: CPU: 1 PID: 15774 at net/ipv4/inet_connection_sock.c:1157 sock_put include/net/sock.h:1976 [inline] WARNING: CPU: 1 PID: 15774 at net/ipv4/inet_connection_sock.c:1157 inet_csk_destroy_sock+0x24b/0x380 net/ipv4/inet_connection_sock.c:1169 Modules linked in: CPU: 1 PID: 15774 Comm: syz-executor.0 Not tainted 6.0.0-rc1-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022 RIP: 0010:inet_csk_destroy_sock+0x24b/0x380 net/ipv4/inet_connection_sock.c:1157 Code: ff ff ff ff f0 0f c1 85 80 00 00 00 83 f8 01 74 0b 85 c0 7e 1c 5b 5d 41 5c 41 5d c3 5b 48 89 ef 5d 41 5c 41 5d e9 d5 7e 7f ff <0f> 0b e9 b9 fe ff ff 5b 4c 89 e7 be 03 00 00 00 5d 41 5c 41 5d e9 RSP: 0018:ffffc9000586fcf8 EFLAGS: 00010246 RAX: dffffc0000000000 RBX: 0000000000000007 RCX: ffffffff877e4a15 RDX: 1ffff11029670eec RSI: 0000000000000008 RDI: ffff88814b387760 RBP: ffff88814b3870c0 R08: 0000000000000000 R09: ffff88814b387127 R10: ffffed1029670e24 R11: 0000000000000000 R12: ffff88814b387120 R13: ffff88814b387158 R14: ffff88814b3870c0 R15: ffff88801ff10780 FS: 0000555556e19400(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f8d521a80c0 CR3: 0000000070c0c000 CR4: 00000000003506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: dccp_close+0xa26/0xdd0 net/dccp/proto.c:1060 inet_release+0xf3/0x210 net/ipv4/af_inet.c:428 __sock_release+0xbb/0x270 net/socket.c:650 sock_close+0x13/0x20 net/socket.c:1365 __fput+0x1f5/0x8c0 fs/file_table.c:320 task_work_run+0xc4/0x160 kernel/task_work.c:177 resume_user_mode_work include/linux/resume_user_mode.h:49 [inline] exit_to_user_mode_loop kernel/entry/common.c:169 [inline] exit_to_user_mode_prepare+0x23c/0x250 kernel/entry/common.c:201 __syscall_exit_to_user_mode_work kernel/entry/common.c:283 [inline] syscall_exit_to_user_mode+0x1d/0x50 kernel/entry/common.c:294 do_syscall_64+0x46/0xb0 arch/x86/entry/common.c:86 entry_SYSCALL_64_after_hwframe+0x63/0xcd RIP: 0033:0x7f8d5203df8b Code: 0f 05 48 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8 63 fc ff ff 8b 7c 24 0c 41 89 c0 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 0c e8 a1 fc ff ff 8b 44 RSP: 002b:00007ffef6e3b7a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003 RAX: 0000000000000000 RBX: 0000000000000007 RCX: 00007f8d5203df8b RDX: 00007f8d51c00ce8 RSI: ffffffffffffffff RDI: 0000000000000006 RBP: 00007f8d521ad980 R08: 0000000000000000 R09: 00007f8d51c00000 R10: 00007f8d51c00cf0 R11: 0000000000000293 R12: 00000000000261fe R13: 00007ffef6e3b8a0 R14: 00007f8d521abf80 R15: 0000000000000032