bisecting cause commit starting from d0f418516022c32ecceaf4275423e5bd3f8743a9 building syzkaller on 8eda0b957e5b39c0c525e74f51d6b39ab8c5b1ac testing commit d0f418516022c32ecceaf4275423e5bd3f8743a9 with gcc (GCC) 8.1.0 kernel signature: 84400f64476f126367c28b7f93b3fdd6853336bb all runs: crashed: WARNING in bpf_warn_invalid_xdp_action testing release v5.4 testing commit 219d54332a09e8d8741c1e1982f5eae56099de85 with gcc (GCC) 8.1.0 kernel signature: 4a907e2b977a06452351617d282489e9ebd7569d all runs: crashed: WARNING in bpf_warn_invalid_xdp_action testing release v5.3 testing commit 4d856f72c10ecb060868ed10ff1b1453943fc6c8 with gcc (GCC) 8.1.0 kernel signature: dad03402abd65a026172703c2cde9dbfe4167e3f all runs: crashed: WARNING in bpf_warn_invalid_xdp_action testing release v5.2 testing commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 with gcc (GCC) 8.1.0 kernel signature: 3cef36b9e4b7dccf379e0f3edf286d92d09451fe all runs: crashed: WARNING in bpf_warn_invalid_xdp_action testing release v5.1 testing commit e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd with gcc (GCC) 8.1.0 kernel signature: d195d789b9eba3c521fdcf269ddab2d5acb094f9 all runs: crashed: WARNING in bpf_warn_invalid_xdp_action testing release v5.0 testing commit 1c163f4c7b3f621efff9b28a47abb36f7378d783 with gcc (GCC) 8.1.0 kernel signature: 2e455cc4e7e53149f027933ca77afad108755efc all runs: crashed: WARNING in bpf_warn_invalid_xdp_action testing release v4.20 testing commit 8fe28cb58bcb235034b64cbbb7550a8a43fd88be with gcc (GCC) 8.1.0 kernel signature: be40af31a31f318a8b5a0f97e778bb150a751c78 all runs: OK # git bisect start 1c163f4c7b3f621efff9b28a47abb36f7378d783 8fe28cb58bcb235034b64cbbb7550a8a43fd88be Bisecting: 7011 revisions left to test after this (roughly 13 steps) [af7ddd8a627c62a835524b3f5b471edbbbcce025] Merge tag 'dma-mapping-4.21' of git://git.infradead.org/users/hch/dma-mapping testing commit af7ddd8a627c62a835524b3f5b471edbbbcce025 with gcc (GCC) 8.1.0 kernel signature: c116d349d51b09bd1e91008ccee11ccb1a416ddf all runs: crashed: WARNING in bpf_warn_invalid_xdp_action # git bisect bad af7ddd8a627c62a835524b3f5b471edbbbcce025 Bisecting: 3448 revisions left to test after this (roughly 12 steps) [792bf4d871dea8b69be2aaabdd320d7c6ed15985] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 792bf4d871dea8b69be2aaabdd320d7c6ed15985 with gcc (GCC) 8.1.0 kernel signature: d7eb95e27d2b9f56e9ad7bc5ad783674a1c99f76 all runs: OK # git bisect good 792bf4d871dea8b69be2aaabdd320d7c6ed15985 Bisecting: 1729 revisions left to test after this (roughly 11 steps) [aa9d6e0f33aea8a1879e7e53fe0e436943f9ce0c] linux/netlink.h: drop unnecessary extern prefix testing commit aa9d6e0f33aea8a1879e7e53fe0e436943f9ce0c with gcc (GCC) 8.1.0 kernel signature: 133c49bd60261e84beb6dfc2c6f9fe993f5a6b0e all runs: crashed: WARNING in bpf_warn_invalid_xdp_action # git bisect bad aa9d6e0f33aea8a1879e7e53fe0e436943f9ce0c Bisecting: 858 revisions left to test after this (roughly 10 steps) [2a95471c3397734ba6869ca3fa084490fb35b40b] Merge branch 'prog_test_run-improvement' testing commit 2a95471c3397734ba6869ca3fa084490fb35b40b with gcc (GCC) 8.1.0 kernel signature: 19a94d304cee2d596fd14e3650e65bd892183022 all runs: OK # git bisect good 2a95471c3397734ba6869ca3fa084490fb35b40b Bisecting: 412 revisions left to test after this (roughly 9 steps) [addb0679839a1f74da6ec742137558be244dd0e9] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next testing commit addb0679839a1f74da6ec742137558be244dd0e9 with gcc (GCC) 8.1.0 kernel signature: 329e38311c723d11d89559f7e83e50ed1f80903a all runs: crashed: WARNING in bpf_warn_invalid_xdp_action # git bisect bad addb0679839a1f74da6ec742137558be244dd0e9 Bisecting: 222 revisions left to test after this (roughly 8 steps) [b6f153d3e5a5bfbda17b997bd9e258143aa11809] selftests: mlxsw: Add one-armed router test testing commit b6f153d3e5a5bfbda17b997bd9e258143aa11809 with gcc (GCC) 8.1.0 kernel signature: 0c9b2e37253560ce68ecacd4cbf6abdfba096d05 all runs: OK # git bisect good b6f153d3e5a5bfbda17b997bd9e258143aa11809 Bisecting: 119 revisions left to test after this (roughly 7 steps) [d8ed257f313f64e9835e61d1365dea95a0a1c9c6] tcp: handle EOR and FIN conditions the same in tcp_tso_should_defer() testing commit d8ed257f313f64e9835e61d1365dea95a0a1c9c6 with gcc (GCC) 8.1.0 kernel signature: 3be15bee4155c3a1cb3bb57b3bacaf283f5181be run #0: crashed: BUG: corrupted list in ___neigh_create run #1: crashed: BUG: corrupted list in neigh_mark_dead run #2: crashed: KASAN: use-after-free Read in neigh_mark_dead run #3: crashed: BUG: corrupted list in corrupted run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: crashed: KASAN: use-after-free Read in neigh_mark_dead # git bisect bad d8ed257f313f64e9835e61d1365dea95a0a1c9c6 Bisecting: 51 revisions left to test after this (roughly 6 steps) [6b241e411607a8f78bee74b96655e9d7835ea8ba] Merge branch 'net-aquantia-add-RSS-configuration' testing commit 6b241e411607a8f78bee74b96655e9d7835ea8ba with gcc (GCC) 8.1.0 kernel signature: 20c821a9f96e22467bbadb66d2aebcdcefff46fe all runs: OK # git bisect good 6b241e411607a8f78bee74b96655e9d7835ea8ba Bisecting: 25 revisions left to test after this (roughly 5 steps) [c3529177db471b964fbe327ffb801266f0482d64] net: hns3: handle hw errors of SSU testing commit c3529177db471b964fbe327ffb801266f0482d64 with gcc (GCC) 8.1.0 kernel signature: 346200444d5805e0eb37fb0116299f4199324e39 all runs: OK # git bisect good c3529177db471b964fbe327ffb801266f0482d64 Bisecting: 12 revisions left to test after this (roughly 4 steps) [120d633f199b264e0ad4c98eedc0564a89171c1d] Merge branch 'platform-data-controls-for-mdio-gpio' testing commit 120d633f199b264e0ad4c98eedc0564a89171c1d with gcc (GCC) 8.1.0 kernel signature: 494044221a42cefeed6e82b024b885d2d74f8787 run #0: crashed: BUG: corrupted list in ___neigh_create run #1: crashed: BUG: corrupted list in neigh_mark_dead run #2: crashed: BUG: corrupted list in neigh_mark_dead run #3: crashed: BUG: corrupted list in neigh_mark_dead run #4: crashed: KASAN: use-after-free Read in neigh_mark_dead run #5: crashed: BUG: corrupted list in ___neigh_create run #6: crashed: BUG: corrupted list in neigh_mark_dead run #7: crashed: BUG: corrupted list in neigh_mark_dead run #8: OK run #9: OK # git bisect bad 120d633f199b264e0ad4c98eedc0564a89171c1d Bisecting: 6 revisions left to test after this (roughly 3 steps) [dfe465d33e7fce145746d836ddb31f0e27f28e28] tc-testing: Add new TdcResults module testing commit dfe465d33e7fce145746d836ddb31f0e27f28e28 with gcc (GCC) 8.1.0 kernel signature: a0aa531697b2c5baf40171f7efaf254e89faf122 run #0: crashed: KASAN: use-after-free Read in neigh_mark_dead run #1: crashed: KASAN: use-after-free Read in neigh_mark_dead run #2: crashed: BUG: corrupted list in neigh_mark_dead run #3: crashed: BUG: corrupted list in ___neigh_create run #4: crashed: KASAN: use-after-free Read in neigh_mark_dead run #5: crashed: KASAN: use-after-free Read in neigh_mark_dead run #6: crashed: BUG: corrupted list in neigh_mark_dead run #7: crashed: BUG: corrupted list in neigh_mark_dead run #8: crashed: BUG: corrupted list in neigh_mark_dead run #9: OK # git bisect bad dfe465d33e7fce145746d836ddb31f0e27f28e28 Bisecting: 2 revisions left to test after this (roughly 2 steps) [58956317c8de52009d1a38a721474c24aef74fe7] neighbor: Improve garbage collection testing commit 58956317c8de52009d1a38a721474c24aef74fe7 with gcc (GCC) 8.1.0 kernel signature: 2475e16495727de236b8d44423549ee101052d4e run #0: crashed: BUG: corrupted list in neigh_mark_dead run #1: crashed: BUG: corrupted list in neigh_mark_dead run #2: crashed: BUG: corrupted list in neigh_mark_dead run #3: crashed: KASAN: use-after-free Read in neigh_mark_dead run #4: crashed: BUG: corrupted list in neigh_mark_dead run #5: crashed: KASAN: use-after-free Read in neigh_mark_dead run #6: crashed: KASAN: use-after-free Read in neigh_mark_dead run #7: crashed: BUG: corrupted list in neigh_mark_dead run #8: OK run #9: OK # git bisect bad 58956317c8de52009d1a38a721474c24aef74fe7 Bisecting: 0 revisions left to test after this (roughly 1 step) [12edfdfc79860f42fa493f81518e040376b6a5bc] Merge branch 'hns3-error-handling' testing commit 12edfdfc79860f42fa493f81518e040376b6a5bc with gcc (GCC) 8.1.0 kernel signature: 2bac5819b833d2cf3bf9a0810680bbed84c49ea8 all runs: OK # git bisect good 12edfdfc79860f42fa493f81518e040376b6a5bc 58956317c8de52009d1a38a721474c24aef74fe7 is the first bad commit commit 58956317c8de52009d1a38a721474c24aef74fe7 Author: David Ahern Date: Fri Dec 7 12:24:57 2018 -0800 neighbor: Improve garbage collection The existing garbage collection algorithm has a number of problems: 1. The gc algorithm will not evict PERMANENT entries as those entries are managed by userspace, yet the existing algorithm walks the entire hash table which means it always considers PERMANENT entries when looking for entries to evict. In some use cases (e.g., EVPN) there can be tens of thousands of PERMANENT entries leading to wasted CPU cycles when gc kicks in. As an example, with 32k permanent entries, neigh_alloc has been observed taking more than 4 msec per invocation. 2. Currently, when the number of neighbor entries hits gc_thresh2 and the last flush for the table was more than 5 seconds ago gc kicks in walks the entire hash table evicting *all* entries not in PERMANENT or REACHABLE state and not marked as externally learned. There is no discriminator on when the neigh entry was created or if it just moved from REACHABLE to another NUD_VALID state (e.g., NUD_STALE). It is possible for entries to be created or for established neighbor entries to be moved to STALE (e.g., an external node sends an ARP request) right before the 5 second window lapses: -----|---------x|----------|----- t-5 t t+5 If that happens those entries are evicted during gc causing unnecessary thrashing on neighbor entries and userspace caches trying to track them. Further, this contradicts the description of gc_thresh2 which says "Entries older than 5 seconds will be cleared". One workaround is to make gc_thresh2 == gc_thresh3 but that negates the whole point of having separate thresholds. 3. Clearing *all* neigh non-PERMANENT/REACHABLE/externally learned entries when gc_thresh2 is exceeded is over kill and contributes to trashing especially during startup. This patch addresses these problems as follows: 1. Use of a separate list_head to track entries that can be garbage collected along with a separate counter. PERMANENT entries are not added to this list. The gc_thresh parameters are only compared to the new counter, not the total entries in the table. The forced_gc function is updated to only walk this new gc_list looking for entries to evict. 2. Entries are added to the list head at the tail and removed from the front. 3. Entries are only evicted if they were last updated more than 5 seconds ago, adhering to the original intent of gc_thresh2. 4. Forced gc is stopped once the number of gc_entries drops below gc_thresh2. 5. Since gc checks do not apply to PERMANENT entries, gc levels are skipped when allocating a new neighbor for a PERMANENT entry. By extension this means there are no explicit limits on the number of PERMANENT entries that can be created, but this is no different than FIB entries or FDB entries. Signed-off-by: David Ahern Signed-off-by: David S. Miller Documentation/networking/ip-sysctl.txt | 4 +- include/net/neighbour.h | 3 + net/core/neighbour.c | 119 +++++++++++++++++++++++---------- 3 files changed, 90 insertions(+), 36 deletions(-) culprit signature: 2475e16495727de236b8d44423549ee101052d4e parent signature: 2bac5819b833d2cf3bf9a0810680bbed84c49ea8 revisions tested: 20, total time: 4h24m58.520145809s (build: 1h58m16.458530283s, test: 2h24m44.757332683s) first bad commit: 58956317c8de52009d1a38a721474c24aef74fe7 neighbor: Improve garbage collection cc: ["corbet@lwn.net" "davem@davemloft.net" "dsahern@gmail.com" "linux-doc@vger.kernel.org" "linux-kernel@vger.kernel.org" "netdev@vger.kernel.org"] crash: BUG: corrupted list in neigh_mark_dead A link change request failed with some changes committed already. Interface lo may have been left with an inconsistent configuration, please check. A link change request failed with some changes committed already. Interface lo may have been left with an inconsistent configuration, please check. list_del corruption. next->prev should be ffff88809290b010, but was ffff8880881bd490 ------------[ cut here ]------------ kernel BUG at lib/list_debug.c:56! invalid opcode: 0000 [#1] PREEMPT SMP KASAN CPU: 0 PID: 5 Comm: kworker/0:0 Not tainted 4.20.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events_power_efficient neigh_periodic_work RIP: 0010:__list_del_entry_valid.cold.1+0x37/0x4a lib/list_debug.c:54 Code: e8 51 ec 1a fe 0f 0b 4c 89 ea 48 89 de 48 c7 c7 20 56 87 87 e8 3d ec 1a fe 0f 0b 48 89 de 48 c7 c7 80 57 87 87 e8 2c ec 1a fe <0f> 0b 48 89 de 48 c7 c7 20 57 87 87 e8 1b ec 1a fe 0f 0b 48 89 d9 RSP: 0018:ffff8880aa1ffc88 EFLAGS: 00010282 RAX: 0000000000000054 RBX: ffff88809290b010 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffffffff87875520 RDI: ffffffff8a351ca0 RBP: ffff8880aa1ffca0 R08: ffffed1015d05021 R09: ffffed1015d05020 R10: ffffed1015d05020 R11: ffff8880ae828107 R12: ffffffff8923e5e0 R13: ffffffff8923e5e0 R14: ffff88809290ada8 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fff1d3f5d38 CR3: 000000008a163000 CR4: 00000000001406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __list_del_entry include/linux/list.h:117 [inline] list_del_init include/linux/list.h:159 [inline] neigh_mark_dead+0x86/0x210 net/core/neighbour.c:125 neigh_periodic_work+0x56f/0x870 net/core/neighbour.c:905 process_one_work+0x830/0x1670 kernel/workqueue.c:2153 worker_thread+0x85/0xb60 kernel/workqueue.c:2296 kthread+0x324/0x3e0 kernel/kthread.c:246 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 Modules linked in: ---[ end trace ae5cfb006a285e45 ]--- RIP: 0010:__list_del_entry_valid.cold.1+0x37/0x4a lib/list_debug.c:54 Code: e8 51 ec 1a fe 0f 0b 4c 89 ea 48 89 de 48 c7 c7 20 56 87 87 e8 3d ec 1a fe 0f 0b 48 89 de 48 c7 c7 80 57 87 87 e8 2c ec 1a fe <0f> 0b 48 89 de 48 c7 c7 20 57 87 87 e8 1b ec 1a fe 0f 0b 48 89 d9 RSP: 0018:ffff8880aa1ffc88 EFLAGS: 00010282 RAX: 0000000000000054 RBX: ffff88809290b010 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffffffff87875520 RDI: ffffffff8a351ca0 RBP: ffff8880aa1ffca0 R08: ffffed1015d05021 R09: ffffed1015d05020 R10: ffffed1015d05020 R11: ffff8880ae828107 R12: ffffffff8923e5e0 R13: ffffffff8923e5e0 R14: ffff88809290ada8 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fff1d3f5d38 CR3: 000000008a163000 CR4: 00000000001406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400