bisecting cause commit starting from d96d875ef5dd372f533059a44f98e92de9cf0d42 building syzkaller on 8eda0b957e5b39c0c525e74f51d6b39ab8c5b1ac testing commit d96d875ef5dd372f533059a44f98e92de9cf0d42 with gcc (GCC) 8.1.0 kernel signature: 0006b6bd09b7d9fb015156fb918ed660721661b1 all runs: crashed: BUG: unable to handle kernel paging request in do_xdp_generic testing release v5.4 testing commit 219d54332a09e8d8741c1e1982f5eae56099de85 with gcc (GCC) 8.1.0 kernel signature: adc7965aabb595178d5e002cb7dc7855778a9442 all runs: crashed: BUG: unable to handle kernel paging request in do_xdp_generic testing release v5.3 testing commit 4d856f72c10ecb060868ed10ff1b1453943fc6c8 with gcc (GCC) 8.1.0 kernel signature: b1cd113bdfa4d696c0ebd9e10effce6780eb853e all runs: crashed: BUG: unable to handle kernel paging request in do_xdp_generic testing release v5.2 testing commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 with gcc (GCC) 8.1.0 kernel signature: 12d79242c4707b7b0c7ff86eee966ceefa76b231 all runs: crashed: BUG: unable to handle kernel paging request in do_xdp_generic testing release v5.1 testing commit e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd with gcc (GCC) 8.1.0 kernel signature: 1d109fc54da4076888af3cb6da1c19479447c634 all runs: crashed: BUG: unable to handle kernel paging request in do_xdp_generic testing release v5.0 testing commit 1c163f4c7b3f621efff9b28a47abb36f7378d783 with gcc (GCC) 8.1.0 kernel signature: d41364cbf715026ab32ecc4f03b2ecee315cb232 all runs: crashed: BUG: unable to handle kernel paging request in do_xdp_generic testing release v4.20 testing commit 8fe28cb58bcb235034b64cbbb7550a8a43fd88be with gcc (GCC) 8.1.0 kernel signature: 5cde044130356c57f217084c983f7c7f1bb1e3da all runs: OK # git bisect start 1c163f4c7b3f621efff9b28a47abb36f7378d783 8fe28cb58bcb235034b64cbbb7550a8a43fd88be Bisecting: 7011 revisions left to test after this (roughly 13 steps) [af7ddd8a627c62a835524b3f5b471edbbbcce025] Merge tag 'dma-mapping-4.21' of git://git.infradead.org/users/hch/dma-mapping testing commit af7ddd8a627c62a835524b3f5b471edbbbcce025 with gcc (GCC) 8.1.0 kernel signature: 92c290a65e6978b48814c5b338731dd18016fdfc all runs: crashed: BUG: unable to handle kernel paging request in do_xdp_generic # git bisect bad af7ddd8a627c62a835524b3f5b471edbbbcce025 Bisecting: 3448 revisions left to test after this (roughly 12 steps) [792bf4d871dea8b69be2aaabdd320d7c6ed15985] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 792bf4d871dea8b69be2aaabdd320d7c6ed15985 with gcc (GCC) 8.1.0 kernel signature: 17407ca5a5e31103c94870db05a511fa2451c76b all runs: OK # git bisect good 792bf4d871dea8b69be2aaabdd320d7c6ed15985 Bisecting: 1729 revisions left to test after this (roughly 11 steps) [aa9d6e0f33aea8a1879e7e53fe0e436943f9ce0c] linux/netlink.h: drop unnecessary extern prefix testing commit aa9d6e0f33aea8a1879e7e53fe0e436943f9ce0c with gcc (GCC) 8.1.0 kernel signature: d45505b3cc1a2a5ebaf2c436d591a79d457584df all runs: crashed: BUG: unable to handle kernel paging request in do_xdp_generic # git bisect bad aa9d6e0f33aea8a1879e7e53fe0e436943f9ce0c Bisecting: 858 revisions left to test after this (roughly 10 steps) [2a95471c3397734ba6869ca3fa084490fb35b40b] Merge branch 'prog_test_run-improvement' testing commit 2a95471c3397734ba6869ca3fa084490fb35b40b with gcc (GCC) 8.1.0 kernel signature: 6e9fcaf2ec20cedd140ba3daca00a88ad3fd3771 all runs: OK # git bisect good 2a95471c3397734ba6869ca3fa084490fb35b40b Bisecting: 412 revisions left to test after this (roughly 9 steps) [addb0679839a1f74da6ec742137558be244dd0e9] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next testing commit addb0679839a1f74da6ec742137558be244dd0e9 with gcc (GCC) 8.1.0 kernel signature: e521c61171cf8e7ea4ea3a29395b8c0a237b4ae3 all runs: crashed: BUG: unable to handle kernel paging request in do_xdp_generic # git bisect bad addb0679839a1f74da6ec742137558be244dd0e9 Bisecting: 222 revisions left to test after this (roughly 8 steps) [b6f153d3e5a5bfbda17b997bd9e258143aa11809] selftests: mlxsw: Add one-armed router test testing commit b6f153d3e5a5bfbda17b997bd9e258143aa11809 with gcc (GCC) 8.1.0 kernel signature: 5d2a00545b481ab0d2a94a6300dee441d7166870 all runs: OK # git bisect good b6f153d3e5a5bfbda17b997bd9e258143aa11809 Bisecting: 119 revisions left to test after this (roughly 7 steps) [d8ed257f313f64e9835e61d1365dea95a0a1c9c6] tcp: handle EOR and FIN conditions the same in tcp_tso_should_defer() testing commit d8ed257f313f64e9835e61d1365dea95a0a1c9c6 with gcc (GCC) 8.1.0 kernel signature: 6169d72d05ff842080fad593b96101dcf4301020 run #0: crashed: BUG: corrupted list in ___neigh_create run #1: crashed: BUG: corrupted list in neigh_mark_dead run #2: crashed: BUG: corrupted list in neigh_mark_dead run #3: crashed: BUG: corrupted list in neigh_mark_dead run #4: crashed: BUG: corrupted list in neigh_mark_dead run #5: crashed: BUG: corrupted list in neigh_mark_dead run #6: OK run #7: OK run #8: OK run #9: crashed: BUG: corrupted list in neigh_mark_dead # git bisect bad d8ed257f313f64e9835e61d1365dea95a0a1c9c6 Bisecting: 51 revisions left to test after this (roughly 6 steps) [6b241e411607a8f78bee74b96655e9d7835ea8ba] Merge branch 'net-aquantia-add-RSS-configuration' testing commit 6b241e411607a8f78bee74b96655e9d7835ea8ba with gcc (GCC) 8.1.0 kernel signature: 4f0aa2e83f3a8dc7bbe8c8a1a3851c16f6e852b5 all runs: OK # git bisect good 6b241e411607a8f78bee74b96655e9d7835ea8ba Bisecting: 25 revisions left to test after this (roughly 5 steps) [c3529177db471b964fbe327ffb801266f0482d64] net: hns3: handle hw errors of SSU testing commit c3529177db471b964fbe327ffb801266f0482d64 with gcc (GCC) 8.1.0 kernel signature: de6bdfb324f11ce4cb48fa39e71979e8678ecb74 all runs: OK # git bisect good c3529177db471b964fbe327ffb801266f0482d64 Bisecting: 12 revisions left to test after this (roughly 4 steps) [120d633f199b264e0ad4c98eedc0564a89171c1d] Merge branch 'platform-data-controls-for-mdio-gpio' testing commit 120d633f199b264e0ad4c98eedc0564a89171c1d with gcc (GCC) 8.1.0 kernel signature: e9fe5db808b94a316808ac900d9b7c9822af3f43 run #0: crashed: BUG: corrupted list in ___neigh_create run #1: crashed: KASAN: use-after-free Read in neigh_mark_dead run #2: crashed: BUG: corrupted list in neigh_mark_dead run #3: crashed: BUG: corrupted list in neigh_mark_dead run #4: crashed: BUG: corrupted list in neigh_mark_dead run #5: crashed: BUG: corrupted list in neigh_mark_dead run #6: OK run #7: OK run #8: OK run #9: crashed: BUG: corrupted list in neigh_mark_dead # git bisect bad 120d633f199b264e0ad4c98eedc0564a89171c1d Bisecting: 6 revisions left to test after this (roughly 3 steps) [dfe465d33e7fce145746d836ddb31f0e27f28e28] tc-testing: Add new TdcResults module testing commit dfe465d33e7fce145746d836ddb31f0e27f28e28 with gcc (GCC) 8.1.0 kernel signature: 6d6c37db35127b09919e6ac1c898a44c0a4b3b9f run #0: crashed: BUG: corrupted list in neigh_mark_dead run #1: crashed: BUG: corrupted list in neigh_mark_dead run #2: crashed: KASAN: use-after-free Read in neigh_mark_dead run #3: crashed: BUG: corrupted list in neigh_mark_dead run #4: crashed: KASAN: use-after-free Read in neigh_mark_dead run #5: crashed: KASAN: use-after-free Read in neigh_mark_dead run #6: OK run #7: crashed: KASAN: use-after-free Read in neigh_mark_dead run #8: OK run #9: OK # git bisect bad dfe465d33e7fce145746d836ddb31f0e27f28e28 Bisecting: 2 revisions left to test after this (roughly 2 steps) [58956317c8de52009d1a38a721474c24aef74fe7] neighbor: Improve garbage collection testing commit 58956317c8de52009d1a38a721474c24aef74fe7 with gcc (GCC) 8.1.0 kernel signature: 380ab9f9034c228db715e1575f1553c74ac7fc10 run #0: crashed: BUG: corrupted list in neigh_mark_dead run #1: crashed: BUG: corrupted list in neigh_mark_dead run #2: crashed: KASAN: use-after-free Read in neigh_mark_dead run #3: crashed: BUG: corrupted list in neigh_mark_dead run #4: crashed: BUG: corrupted list in neigh_mark_dead run #5: OK run #6: OK run #7: OK run #8: crashed: BUG: corrupted list in neigh_mark_dead run #9: crashed: BUG: corrupted list in neigh_mark_dead # git bisect bad 58956317c8de52009d1a38a721474c24aef74fe7 Bisecting: 0 revisions left to test after this (roughly 1 step) [12edfdfc79860f42fa493f81518e040376b6a5bc] Merge branch 'hns3-error-handling' testing commit 12edfdfc79860f42fa493f81518e040376b6a5bc with gcc (GCC) 8.1.0 kernel signature: ba7ed456653de5313e2f9c88a1441f28167df440 all runs: OK # git bisect good 12edfdfc79860f42fa493f81518e040376b6a5bc 58956317c8de52009d1a38a721474c24aef74fe7 is the first bad commit commit 58956317c8de52009d1a38a721474c24aef74fe7 Author: David Ahern Date: Fri Dec 7 12:24:57 2018 -0800 neighbor: Improve garbage collection The existing garbage collection algorithm has a number of problems: 1. The gc algorithm will not evict PERMANENT entries as those entries are managed by userspace, yet the existing algorithm walks the entire hash table which means it always considers PERMANENT entries when looking for entries to evict. In some use cases (e.g., EVPN) there can be tens of thousands of PERMANENT entries leading to wasted CPU cycles when gc kicks in. As an example, with 32k permanent entries, neigh_alloc has been observed taking more than 4 msec per invocation. 2. Currently, when the number of neighbor entries hits gc_thresh2 and the last flush for the table was more than 5 seconds ago gc kicks in walks the entire hash table evicting *all* entries not in PERMANENT or REACHABLE state and not marked as externally learned. There is no discriminator on when the neigh entry was created or if it just moved from REACHABLE to another NUD_VALID state (e.g., NUD_STALE). It is possible for entries to be created or for established neighbor entries to be moved to STALE (e.g., an external node sends an ARP request) right before the 5 second window lapses: -----|---------x|----------|----- t-5 t t+5 If that happens those entries are evicted during gc causing unnecessary thrashing on neighbor entries and userspace caches trying to track them. Further, this contradicts the description of gc_thresh2 which says "Entries older than 5 seconds will be cleared". One workaround is to make gc_thresh2 == gc_thresh3 but that negates the whole point of having separate thresholds. 3. Clearing *all* neigh non-PERMANENT/REACHABLE/externally learned entries when gc_thresh2 is exceeded is over kill and contributes to trashing especially during startup. This patch addresses these problems as follows: 1. Use of a separate list_head to track entries that can be garbage collected along with a separate counter. PERMANENT entries are not added to this list. The gc_thresh parameters are only compared to the new counter, not the total entries in the table. The forced_gc function is updated to only walk this new gc_list looking for entries to evict. 2. Entries are added to the list head at the tail and removed from the front. 3. Entries are only evicted if they were last updated more than 5 seconds ago, adhering to the original intent of gc_thresh2. 4. Forced gc is stopped once the number of gc_entries drops below gc_thresh2. 5. Since gc checks do not apply to PERMANENT entries, gc levels are skipped when allocating a new neighbor for a PERMANENT entry. By extension this means there are no explicit limits on the number of PERMANENT entries that can be created, but this is no different than FIB entries or FDB entries. Signed-off-by: David Ahern Signed-off-by: David S. Miller Documentation/networking/ip-sysctl.txt | 4 +- include/net/neighbour.h | 3 + net/core/neighbour.c | 119 +++++++++++++++++++++++---------- 3 files changed, 90 insertions(+), 36 deletions(-) culprit signature: 380ab9f9034c228db715e1575f1553c74ac7fc10 parent signature: ba7ed456653de5313e2f9c88a1441f28167df440 revisions tested: 20, total time: 4h29m39.573723461s (build: 2h3m26.756076548s, test: 2h24m14.970044872s) first bad commit: 58956317c8de52009d1a38a721474c24aef74fe7 neighbor: Improve garbage collection cc: ["corbet@lwn.net" "davem@davemloft.net" "dsahern@gmail.com" "linux-doc@vger.kernel.org" "linux-kernel@vger.kernel.org" "netdev@vger.kernel.org"] crash: BUG: corrupted list in neigh_mark_dead device veth0_macvtap left promiscuous mode device veth1_vlan left promiscuous mode device veth0_vlan left promiscuous mode list_del corruption. prev->next should be ffff888089392bd0, but was ffff88808e59cb90 ------------[ cut here ]------------ kernel BUG at lib/list_debug.c:53! invalid opcode: 0000 [#1] PREEMPT SMP KASAN CPU: 0 PID: 7 Comm: kworker/u4:0 Not tainted 4.20.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: netns cleanup_net RIP: 0010:__list_del_entry_valid.cold.1+0x48/0x4a lib/list_debug.c:51 Code: 56 87 87 e8 3d ec 1a fe 0f 0b 48 89 de 48 c7 c7 80 57 87 87 e8 2c ec 1a fe 0f 0b 48 89 de 48 c7 c7 20 57 87 87 e8 1b ec 1a fe <0f> 0b 48 89 d9 48 c7 c7 e0 57 87 87 e8 0a ec 1a fe 0f 0b 48 89 f1 RSP: 0018:ffff8880aa21f490 EFLAGS: 00010286 RAX: 0000000000000054 RBX: ffff888089392bd0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffffffff87875520 RDI: ffffffff8a351ca0 RBP: ffff8880aa21f4a8 R08: ffffed1015d05021 R09: ffffed1015d05020 R10: ffffed1015d05020 R11: ffff8880ae828107 R12: ffff88808e612750 R13: ffff888096fd8750 R14: ffff888089392940 R15: dffffc0000000000 FS: 0000000000000000(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffff600400 CR3: 00000000a8075000 CR4: 00000000001406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __list_del_entry include/linux/list.h:117 [inline] list_del_init include/linux/list.h:159 [inline] neigh_mark_dead+0x86/0x210 net/core/neighbour.c:125 neigh_flush_dev+0x232/0x620 net/core/neighbour.c:290 __neigh_ifdown+0x46/0x310 net/core/neighbour.c:328 neigh_ifdown+0xb/0x10 net/core/neighbour.c:345 rt6_disable_ip+0x3f5/0x4b0 net/ipv6/route.c:4046 addrconf_ifdown+0x7a/0xf60 net/ipv6/addrconf.c:3669 addrconf_notify+0x531/0x1fa0 net/ipv6/addrconf.c:3594 notifier_call_chain+0x8a/0x160 kernel/notifier.c:93 __raw_notifier_call_chain kernel/notifier.c:394 [inline] raw_notifier_call_chain+0x11/0x20 kernel/notifier.c:401 call_netdevice_notifiers_info+0x28/0x60 net/core/dev.c:1737 call_netdevice_notifiers_extack net/core/dev.c:1749 [inline] call_netdevice_notifiers net/core/dev.c:1763 [inline] dev_close_many+0x2c6/0x640 net/core/dev.c:1507 rollback_registered_many+0x36d/0xcf0 net/core/dev.c:8036 unregister_netdevice_many+0x3e/0x1f0 net/core/dev.c:9164 default_device_exit_batch+0x2e4/0x3d0 net/core/dev.c:9633 ops_exit_list.isra.5+0xd3/0x120 net/core/net_namespace.c:156 cleanup_net+0x363/0x840 net/core/net_namespace.c:551 process_one_work+0x830/0x1670 kernel/workqueue.c:2153 worker_thread+0x85/0xb60 kernel/workqueue.c:2296 kthread+0x324/0x3e0 kernel/kthread.c:246 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 Modules linked in: ---[ end trace 86440e027f74ea15 ]--- RIP: 0010:__list_del_entry_valid.cold.1+0x48/0x4a lib/list_debug.c:51 Code: 56 87 87 e8 3d ec 1a fe 0f 0b 48 89 de 48 c7 c7 80 57 87 87 e8 2c ec 1a fe 0f 0b 48 89 de 48 c7 c7 20 57 87 87 e8 1b ec 1a fe <0f> 0b 48 89 d9 48 c7 c7 e0 57 87 87 e8 0a ec 1a fe 0f 0b 48 89 f1 RSP: 0018:ffff8880aa21f490 EFLAGS: 00010286 RAX: 0000000000000054 RBX: ffff888089392bd0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffffffff87875520 RDI: ffffffff8a351ca0 RBP: ffff8880aa21f4a8 R08: ffffed1015d05021 R09: ffffed1015d05020 R10: ffffed1015d05020 R11: ffff8880ae828107 R12: ffff88808e612750 R13: ffff888096fd8750 R14: ffff888089392940 R15: dffffc0000000000 FS: 0000000000000000(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffff600400 CR3: 00000000a8075000 CR4: 00000000001406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400