bisecting cause commit starting from 2fcd14d0f78090f57aecd7f424e2b0373cd631a7 building syzkaller on 8cac236e8c3741446e540b2fe0702086a4ae4c17 testing commit 2fcd14d0f78090f57aecd7f424e2b0373cd631a7 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: af01fd083f795d22d349a408b32d2d50925be35d6858d34e8a08e8bf020e1a65 all runs: crashed: WARNING in __dev_set_promiscuity testing release v5.14 testing commit 7d2a07b769330c34b4deabeed939325c77a7ec2f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 8c219514f33cf5f7e995cffb6643e798eb9f0e406da2dbcbb7423680d5a3f503 all runs: OK # git bisect start 2fcd14d0f78090f57aecd7f424e2b0373cd631a7 7d2a07b769330c34b4deabeed939325c77a7ec2f Bisecting: 6435 revisions left to test after this (roughly 13 steps) [1b4f3dfb4792f03b139edf10124fcbeb44e608e6] Merge tag 'usb-serial-5.15-rc1' of https://git.kernel.org/pub/scm/linux/kernel/git/johan/usb-serial into usb-next testing commit 1b4f3dfb4792f03b139edf10124fcbeb44e608e6 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2a21ff9c3775a869ed875a487dbb63bb481a58752b4a2427f39fc52291f831d7 run #0: crashed: WARNING in __dev_set_promiscuity run #1: crashed: WARNING in __dev_set_promiscuity run #2: crashed: WARNING in __dev_set_promiscuity run #3: crashed: WARNING in __dev_set_promiscuity run #4: crashed: WARNING in __dev_set_promiscuity run #5: crashed: WARNING in __dev_set_promiscuity run #6: crashed: WARNING in __dev_set_promiscuity run #7: crashed: WARNING in __dev_set_promiscuity run #8: crashed: WARNING in __dev_set_promiscuity run #9: boot failed: possible deadlock in blktrans_open # git bisect bad 1b4f3dfb4792f03b139edf10124fcbeb44e608e6 Bisecting: 3531 revisions left to test after this (roughly 11 steps) [29ce8f9701072fc221d9c38ad952de1a9578f95c] Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net testing commit 29ce8f9701072fc221d9c38ad952de1a9578f95c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: cfe4b10c44e94c036f527eb6acfd0e3fa09eb6e2154ec551d98ca2e2bfebf244 all runs: crashed: WARNING in __dev_set_promiscuity # git bisect bad 29ce8f9701072fc221d9c38ad952de1a9578f95c Bisecting: 951 revisions left to test after this (roughly 10 steps) [bed5a942e27e1df67250e27e1f2eb5ea2d4cc362] Merge tag 'mlx5-updates-2021-08-11' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux testing commit bed5a942e27e1df67250e27e1f2eb5ea2d4cc362 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: b24be650869c28ac0dd364378e7949d9c8dea544593012884dac7a6382a035d3 all runs: OK # git bisect good bed5a942e27e1df67250e27e1f2eb5ea2d4cc362 Bisecting: 475 revisions left to test after this (roughly 9 steps) [38cbd6e77f85c7fbf5a34a0aebf9c3e6d01fc214] Merge branch 'lan7800-improvements' testing commit 38cbd6e77f85c7fbf5a34a0aebf9c3e6d01fc214 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 9b4987961562c63b5b1bcb10ca7ef5d4ddac968dfa305d2c36ad39c55f680fdd all runs: crashed: WARNING in __dev_set_promiscuity # git bisect bad 38cbd6e77f85c7fbf5a34a0aebf9c3e6d01fc214 Bisecting: 215 revisions left to test after this (roughly 8 steps) [e61fbee7be4b9566f8e8fcb15aadad0f17936c31] Merge tag 'for-net-next-2021-08-19' of git://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next testing commit e61fbee7be4b9566f8e8fcb15aadad0f17936c31 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c7612447fec7e6b7479d17df7134e94e1df4a72b6319fc9544fb67c906e26a77 all runs: OK # git bisect good e61fbee7be4b9566f8e8fcb15aadad0f17936c31 Bisecting: 107 revisions left to test after this (roughly 7 steps) [82e34c8a9bdfa80b046a74f09f8f77c17e3e82c8] Revert "Revert "cxgb4: Search VPD with pci_vpd_find_ro_info_keyword()"" testing commit 82e34c8a9bdfa80b046a74f09f8f77c17e3e82c8 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6b1e08eb188c4d75d337b9b3b4d7093400f42ae8027d1da939dabb4348bcb6e0 all runs: OK # git bisect good 82e34c8a9bdfa80b046a74f09f8f77c17e3e82c8 Bisecting: 49 revisions left to test after this (roughly 6 steps) [e6a70a02defd9200ed4b9fbf3714fa9622a93fa3] Merge tag 'wireless-drivers-next-2021-08-22' of git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next testing commit e6a70a02defd9200ed4b9fbf3714fa9622a93fa3 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 173c76c240c2de0e9f21c24b0d91544907c16cb3305a5418dad6b20c81d0d056 all runs: OK # git bisect good e6a70a02defd9200ed4b9fbf3714fa9622a93fa3 Bisecting: 24 revisions left to test after this (roughly 5 steps) [95d1d2490c278ea316a4350f4c24818275fb989c] netdevice: move xdp_rxq within netdev_rx_queue testing commit 95d1d2490c278ea316a4350f4c24818275fb989c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c0056013b4a00e7d984763b29c2fbe7bca659edc1ebd05c095a89222a6024437 all runs: OK # git bisect good 95d1d2490c278ea316a4350f4c24818275fb989c Bisecting: 12 revisions left to test after this (roughly 4 steps) [21631d2d741a64a073e167c27769e73bc7844a2f] xen/netfront: disentangle tx_skb_freelist testing commit 21631d2d741a64a073e167c27769e73bc7844a2f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 9df67cfa7690a1e30d24ced04947511a38df817192e65bdc6a5466fc567acba1 all runs: crashed: WARNING in __dev_set_promiscuity # git bisect bad 21631d2d741a64a073e167c27769e73bc7844a2f Bisecting: 5 revisions left to test after this (roughly 3 steps) [7844ec21a915cc60f1e2cd8682b943b916a7d2fc] selftests/net: Use kselftest skip code for skipped tests testing commit 7844ec21a915cc60f1e2cd8682b943b916a7d2fc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 77d6ec63d6bf5adf85f5e29198ab79de47715abf425c31f10e075a3c427bdcc2 all runs: OK # git bisect good 7844ec21a915cc60f1e2cd8682b943b916a7d2fc Bisecting: 2 revisions left to test after this (roughly 2 steps) [755f905340802e2efb1ea96b9ce0f442916c2645] qed: Enable automatic recovery on error condition. testing commit 755f905340802e2efb1ea96b9ce0f442916c2645 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 9df67cfa7690a1e30d24ced04947511a38df817192e65bdc6a5466fc567acba1 all runs: crashed: WARNING in __dev_set_promiscuity # git bisect bad 755f905340802e2efb1ea96b9ce0f442916c2645 Bisecting: 0 revisions left to test after this (roughly 1 step) [406f42fa0d3cbcea3766c3111d79ac5afe711c5b] net-next: When a bond have a massive amount of VLANs with IPv6 addresses, performance of changing link state, attaching a VRF, changing an IPv6 address, etc. go down dramtically. testing commit 406f42fa0d3cbcea3766c3111d79ac5afe711c5b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 9df67cfa7690a1e30d24ced04947511a38df817192e65bdc6a5466fc567acba1 all runs: crashed: WARNING in __dev_set_promiscuity # git bisect bad 406f42fa0d3cbcea3766c3111d79ac5afe711c5b Bisecting: 0 revisions left to test after this (roughly 0 steps) [a37c5c26693eadb3aa4101d8fe955e40d206b386] net: bridge: change return type of br_handle_ingress_vlan_tunnel testing commit a37c5c26693eadb3aa4101d8fe955e40d206b386 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2af134752e85c85461464aaaed6a9d643b1d9ec02aeea5d0474811b23158c9b6 all runs: OK # git bisect good a37c5c26693eadb3aa4101d8fe955e40d206b386 406f42fa0d3cbcea3766c3111d79ac5afe711c5b is the first bad commit commit 406f42fa0d3cbcea3766c3111d79ac5afe711c5b Author: Gilad Naaman Date: Thu Aug 19 10:17:27 2021 +0300 net-next: When a bond have a massive amount of VLANs with IPv6 addresses, performance of changing link state, attaching a VRF, changing an IPv6 address, etc. go down dramtically. The source of most of the slow down is the `dev_addr_lists.c` module, which mainatins a linked list of HW addresses. When using IPv6, this list grows for each IPv6 address added on a VLAN, since each IPv6 address has a multicast HW address associated with it. When performing any modification to the involved links, this list is traversed many times, often for nothing, all while holding the RTNL lock. Instead, this patch adds an auxilliary rbtree which cuts down traversal time significantly. Performance can be seen with the following script: #!/bin/bash ip netns del test || true 2>/dev/null ip netns add test echo 1 | ip netns exec test tee /proc/sys/net/ipv6/conf/all/keep_addr_on_down > /dev/null set -e ip -n test link add foo type veth peer name bar ip -n test link add b1 type bond ip -n test link add florp type vrf table 10 ip -n test link set bar master b1 ip -n test link set foo up ip -n test link set bar up ip -n test link set b1 up ip -n test link set florp up VLAN_COUNT=1500 BASE_DEV=b1 echo Creating vlans ip netns exec test time -p bash -c "for i in \$(seq 1 $VLAN_COUNT); do ip -n test link add link $BASE_DEV name foo.\$i type vlan id \$i; done" echo Bringing them up ip netns exec test time -p bash -c "for i in \$(seq 1 $VLAN_COUNT); do ip -n test link set foo.\$i up; done" echo Assiging IPv6 Addresses ip netns exec test time -p bash -c "for i in \$(seq 1 $VLAN_COUNT); do ip -n test address add dev foo.\$i 2000::\$i/64; done" echo Attaching to VRF ip netns exec test time -p bash -c "for i in \$(seq 1 $VLAN_COUNT); do ip -n test link set foo.\$i master florp; done" On an Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz machine, the performance before the patch is (truncated): Creating vlans real 108.35 Bringing them up real 4.96 Assiging IPv6 Addresses real 19.22 Attaching to VRF real 458.84 After the patch: Creating vlans real 5.59 Bringing them up real 5.07 Assiging IPv6 Addresses real 5.64 Attaching to VRF real 25.37 Cc: David S. Miller Cc: Jakub Kicinski Cc: Lu Wei Cc: Xiongfeng Wang Cc: Taehee Yoo Signed-off-by: Gilad Naaman Signed-off-by: David S. Miller include/linux/netdevice.h | 5 ++ net/core/dev_addr_lists.c | 144 +++++++++++++++++++++++++++++++--------------- 2 files changed, 103 insertions(+), 46 deletions(-) culprit signature: 9df67cfa7690a1e30d24ced04947511a38df817192e65bdc6a5466fc567acba1 parent signature: 2af134752e85c85461464aaaed6a9d643b1d9ec02aeea5d0474811b23158c9b6 revisions tested: 15, total time: 3h48m2.978143752s (build: 1h42m56.942815147s, test: 2h3m32.168331422s) first bad commit: 406f42fa0d3cbcea3766c3111d79ac5afe711c5b net-next: When a bond have a massive amount of VLANs with IPv6 addresses, performance of changing link state, attaching a VRF, changing an IPv6 address, etc. go down dramtically. recipients (to): ["davem@davemloft.net" "davem@davemloft.net" "gnaaman@drivenets.com" "kuba@kernel.org" "netdev@vger.kernel.org"] recipients (cc): ["ap420073@gmail.com" "linux-kernel@vger.kernel.org" "luwei32@huawei.com" "wangxiongfeng2@huawei.com"] crash: WARNING in __dev_set_promiscuity ------------[ cut here ]------------ RTNL: assertion failed at net/core/dev.c (8529) WARNING: CPU: 0 PID: 12141 at net/core/dev.c:8529 __dev_set_promiscuity+0x14f/0x1a0 net/core/dev.c:8529 Modules linked in: CPU: 0 PID: 12141 Comm: syz-executor.1 Not tainted 5.14.0-rc6-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:__dev_set_promiscuity+0x14f/0x1a0 net/core/dev.c:8529 Code: d9 4b e8 05 00 0f 85 2a ff ff ff ba 51 21 00 00 48 c7 c6 40 02 bd 89 48 c7 c7 80 02 bd 89 c6 05 b9 4b e8 05 01 e8 fc 49 a2 01 <0f> 0b 44 8b 14 24 e9 00 ff ff ff 4c 89 f7 44 89 14 24 e8 fa 4f 26 RSP: 0018:ffffc90002adf2b0 EFLAGS: 00010286 RAX: 0000000000000000 RBX: 00000000ffffffff RCX: 0000000000000000 RDX: 0000000000000203 RSI: ffffffff891fff40 RDI: fffff5200055be48 RBP: ffff88802b82e000 R08: 0000000000000001 R09: ffff8880b9e4bfc7 R10: ffffed10173c97f8 R11: 7361203a4c4e5452 R12: ffff88802b82e338 R13: 0000000000000000 R14: ffff88802b82e0c0 R15: 0000000000001103 FS: 00007f4f5dfb4700(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000200002c0 CR3: 00000000240f4000 CR4: 00000000001506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __dev_set_rx_mode+0x1be/0x250 net/core/dev.c:8672 dev_uc_sync_multiple+0x11b/0x150 net/core/dev_addr_lists.c:732 team_set_rx_mode+0x96/0x1a0 drivers/net/team/team.c:1779 __dev_mc_add net/core/dev_addr_lists.c:830 [inline] dev_mc_add+0xd7/0xf0 net/core/dev_addr_lists.c:844 clusterip_config_init net/ipv4/netfilter/ipt_CLUSTERIP.c:265 [inline] clusterip_tg_check+0xd35/0x1b70 net/ipv4/netfilter/ipt_CLUSTERIP.c:517 xt_check_target+0x1ee/0x910 net/netfilter/x_tables.c:1038 check_target net/ipv4/netfilter/ip_tables.c:511 [inline] find_check_entry.constprop.0+0x6bb/0x920 net/ipv4/netfilter/ip_tables.c:553 translate_table+0xba8/0x1700 net/ipv4/netfilter/ip_tables.c:717 do_replace net/ipv4/netfilter/ip_tables.c:1135 [inline] do_ipt_set_ctl+0x458/0xc00 net/ipv4/netfilter/ip_tables.c:1629 nf_setsockopt+0x6a/0xc0 net/netfilter/nf_sockopt.c:101 ip_setsockopt+0x410/0x2f20 net/ipv4/ip_sockglue.c:1435 tcp_setsockopt+0x10d/0x1e80 net/ipv4/tcp.c:3657 __sys_setsockopt+0x1fd/0x4e0 net/socket.c:2170 __do_sys_setsockopt net/socket.c:2181 [inline] __se_sys_setsockopt net/socket.c:2178 [inline] __x64_sys_setsockopt+0xb5/0x150 net/socket.c:2178 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7f4f5e83d709 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f4f5dfb4188 EFLAGS: 00000246 ORIG_RAX: 0000000000000036 RAX: ffffffffffffffda RBX: 00007f4f5e941f60 RCX: 00007f4f5e83d709 RDX: 0000000000000040 RSI: 0004000000000000 RDI: 0000000000000003 RBP: 00007f4f5e897cb4 R08: 0000000000000298 R09: 0000000000000000 R10: 00000000200002c0 R11: 0000000000000246 R12: 0000000000000000 R13: 00007ffe3c9c5c9f R14: 00007f4f5dfb4300 R15: 0000000000022000