bisecting cause commit starting from de55c9a1967c798fba6d8afe19826f8f03049db3 building syzkaller on 9e8eaa75a18a5cf8102e862be692c0781759e51b testing commit de55c9a1967c798fba6d8afe19826f8f03049db3 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 8f7cdd2bb1d91f03f3b54a1d585b0a58a8267f748e0fca85de2721c36c360421 run #0: basic kernel testing failed: BUG: program execution failed: executor NUM: exit status NUM run #1: crashed: BUG: missing reserved tailroom run #2: crashed: BUG: missing reserved tailroom run #3: crashed: BUG: missing reserved tailroom run #4: crashed: BUG: missing reserved tailroom run #5: crashed: BUG: missing reserved tailroom run #6: crashed: BUG: missing reserved tailroom run #7: crashed: BUG: missing reserved tailroom run #8: crashed: BUG: missing reserved tailroom run #9: crashed: BUG: missing reserved tailroom run #10: crashed: BUG: missing reserved tailroom run #11: crashed: BUG: missing reserved tailroom run #12: crashed: BUG: missing reserved tailroom run #13: crashed: BUG: missing reserved tailroom run #14: crashed: BUG: missing reserved tailroom run #15: crashed: BUG: missing reserved tailroom run #16: crashed: BUG: missing reserved tailroom run #17: crashed: BUG: missing reserved tailroom run #18: crashed: BUG: missing reserved tailroom run #19: crashed: BUG: missing reserved tailroom testing release v5.16 testing commit df0cc57e057f18e44dac8e6c18aba47ab53202f9 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1fd9ab6ec52dc5d6eec160897ae3403fe7938723519a9996a423afd6588ef1fc all runs: OK # git bisect start de55c9a1967c798fba6d8afe19826f8f03049db3 df0cc57e057f18e44dac8e6c18aba47ab53202f9 Bisecting: 7786 revisions left to test after this (roughly 13 steps) [ce990f1de0bc6ff3de43d385e0985efa980fba24] Merge tag 'for-linus-5.17-rc1-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip testing commit ce990f1de0bc6ff3de43d385e0985efa980fba24 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c91a2bd70f49131ef04f6cce82100a5aee19ebfed621ef8a7e7239b3a57d31d9 all runs: OK # git bisect good ce990f1de0bc6ff3de43d385e0985efa980fba24 Bisecting: 3894 revisions left to test after this (roughly 12 steps) [369af20a2c3f738c8610d75b010ee5dc5d3c207f] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi testing commit 369af20a2c3f738c8610d75b010ee5dc5d3c207f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 300a523b60f9e84ad366def2e8e2eac7b4de61575b316ec3cb6c58b85e3e42ab all runs: OK # git bisect good 369af20a2c3f738c8610d75b010ee5dc5d3c207f Bisecting: 1954 revisions left to test after this (roughly 11 steps) [201b5c016f1655ead2ecc7e7e270841782cbbd0b] Merge tag 'efi-urgent-for-v5.17-2' of git://git.kernel.org/pub/scm/linux/kernel/git/efi/efi testing commit 201b5c016f1655ead2ecc7e7e270841782cbbd0b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c42cc419624fbfc777e9876a5caa288c91392cbc3d922bc6b902c4ebff24f60a all runs: OK # git bisect good 201b5c016f1655ead2ecc7e7e270841782cbbd0b Bisecting: 953 revisions left to test after this (roughly 10 steps) [b96a79253fff1cd2c928b379eadd8c7a6f8055e1] Merge tag 'wireless-next-2022-02-11' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next testing commit b96a79253fff1cd2c928b379eadd8c7a6f8055e1 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 86e23a77ea7fbd0f723c540ed4411f3b69a1ae787ee3703090fd710cc16fefcf all runs: basic kernel testing failed: WARNING: suspicious RCU usage in hsr_node_get_first # git bisect skip b96a79253fff1cd2c928b379eadd8c7a6f8055e1 Bisecting: 953 revisions left to test after this (roughly 10 steps) [ed5f9cf06b20f74c1098d6d62313e3e9af217fcb] net/mlx5e: Use READ_ONCE/WRITE_ONCE for DCBX trust state testing commit ed5f9cf06b20f74c1098d6d62313e3e9af217fcb compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 4869a7cfbc4510a81fa752a452ef8538ea89d319ce5208ac99088d56969fc9d9 all runs: basic kernel testing failed: WARNING: suspicious RCU usage in hsr_node_get_first # git bisect skip ed5f9cf06b20f74c1098d6d62313e3e9af217fcb Bisecting: 953 revisions left to test after this (roughly 10 steps) [9ddd1cac6fe1f8464f54ab0d5af9bc8260caca12] mptcp: print out reset infos of MP_RST testing commit 9ddd1cac6fe1f8464f54ab0d5af9bc8260caca12 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2eef781fbd8b48c52efeeabde460dc1ccc8cf6f55791fc0f0e933a29f1ab67f6 all runs: OK # git bisect good 9ddd1cac6fe1f8464f54ab0d5af9bc8260caca12 Bisecting: 780 revisions left to test after this (roughly 10 steps) [a3fc4b1d09d99cdb6a7dbba5a753db15a10b2e9c] Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next testing commit a3fc4b1d09d99cdb6a7dbba5a753db15a10b2e9c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 64084167706ddcb0052682e5e8760a6a7116310fc7ed4ffae8c5f5e3eb86407d all runs: basic kernel testing failed: WARNING: suspicious RCU usage in hsr_node_get_first # git bisect skip a3fc4b1d09d99cdb6a7dbba5a753db15a10b2e9c Bisecting: 780 revisions left to test after this (roughly 10 steps) [b76bc129839d65fa8dbeefd3581dacd54596706f] i40e: Add a stat for tracking busy rx pages testing commit b76bc129839d65fa8dbeefd3581dacd54596706f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: ea3555458ccfaacde9b40dd9e26755ab2ceb22dbd24fb6f4ffa2b08bcfb26dda all runs: basic kernel testing failed: WARNING: suspicious RCU usage in hsr_node_get_first # git bisect skip b76bc129839d65fa8dbeefd3581dacd54596706f Bisecting: 780 revisions left to test after this (roughly 10 steps) [1d21c327281afb19448405dae9ff6b54a9e9bc37] net: ti: am65-cpsw-nuss: remove guards against !BRIDGE_VLAN_INFO_BRENTRY testing commit 1d21c327281afb19448405dae9ff6b54a9e9bc37 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d467ed43cb417d3707c1004ffd585184eb5cb9d4d6e995cb43820e213e2d5ef7 all runs: basic kernel testing failed: WARNING: suspicious RCU usage in hsr_node_get_first # git bisect skip 1d21c327281afb19448405dae9ff6b54a9e9bc37 Bisecting: 780 revisions left to test after this (roughly 10 steps) [c28748233b4736bd31b3d3c3011d42054cc738f5] selftests/bpf: Test BPF_KPROBE_SYSCALL macro testing commit c28748233b4736bd31b3d3c3011d42054cc738f5 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: bb4bc8ad675c7276bd7110f1235507b6281d4159dbe32fc0ad382b10d21b1d40 run #0: basic kernel testing failed: BUG: program execution failed: executor NUM: exit status NUM run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good c28748233b4736bd31b3d3c3011d42054cc738f5 Bisecting: 712 revisions left to test after this (roughly 10 steps) [25bd462fa42f58ca43c881b486726bb81be5aa2b] selftests: fib_test: Add a test case for IPv4 broadcast neighbours testing commit 25bd462fa42f58ca43c881b486726bb81be5aa2b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a690fcb6283c9c3b61e752cd9d19999611919d33bd91dbfb285f3281e6fa9703 run #0: basic kernel testing failed: BUG: program execution failed: executor NUM: exit status NUM run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 25bd462fa42f58ca43c881b486726bb81be5aa2b Bisecting: 367 revisions left to test after this (roughly 9 steps) [3d5985a185e6abfc0b38ed187819016a79eca864] ice: convert VF storage to hash table with krefs and RCU testing commit 3d5985a185e6abfc0b38ed187819016a79eca864 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a1c2894df941badd9003fc91ca7e76914627f0ea64ced5b6d0ca06e41d77b897 all runs: OK # git bisect good 3d5985a185e6abfc0b38ed187819016a79eca864 Bisecting: 183 revisions left to test after this (roughly 8 steps) [c3cde44f3c6e44ce42368066dede0bc3c9b1952a] net: mscc: ocelot: use pretty names for IPPROTO_UDP and IPPROTO_TCP testing commit c3cde44f3c6e44ce42368066dede0bc3c9b1952a compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6189069fad59e6d3930c175acf5f3e94df7188952d7aa4522ecd62eb9429b6a2 all runs: OK # git bisect good c3cde44f3c6e44ce42368066dede0bc3c9b1952a Bisecting: 95 revisions left to test after this (roughly 7 steps) [6646dc241dd09e1e6141b32cd72e5b59c17c0ce5] Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next testing commit 6646dc241dd09e1e6141b32cd72e5b59c17c0ce5 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 14d4541cd27d477dce17efe44b7a5e90fe33ee45778738468095f28f0484b5ee all runs: OK # git bisect good 6646dc241dd09e1e6141b32cd72e5b59c17c0ce5 Bisecting: 47 revisions left to test after this (roughly 6 steps) [0b79b8dc97b9df4f873f63161e3050bafc4c4237] net: axienet: add coalesce timer ethtool configuration testing commit 0b79b8dc97b9df4f873f63161e3050bafc4c4237 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 44eb68ce20e219595911ff658c3fd96bb76e12bb9ba5a3a5a021f37b9dd4dc0d all runs: OK # git bisect good 0b79b8dc97b9df4f873f63161e3050bafc4c4237 Bisecting: 23 revisions left to test after this (roughly 5 steps) [401af75c4975759d45c3627a7588351b4ab3e2a8] Merge branch 'Fixes for bad PTR_TO_BTF_ID offset' testing commit 401af75c4975759d45c3627a7588351b4ab3e2a8 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 40056fea0bed0de69acbb93ac6029475306a75d844952d158dc3bc9468400664 all runs: OK # git bisect good 401af75c4975759d45c3627a7588351b4ab3e2a8 Bisecting: 11 revisions left to test after this (roughly 4 steps) [44e9a741cad824f45112b79c21d88c201d5aec13] bpf: Determine buf_info inside check_buffer_access() testing commit 44e9a741cad824f45112b79c21d88c201d5aec13 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: daca040d1a70bc118b213bbc85678d4df98b3a67b0497c16ac9eefdeed417de4 run #0: basic kernel testing failed: BUG: program execution failed: executor NUM: failed to write control pipe: write |NUM: broken pipe run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 44e9a741cad824f45112b79c21d88c201d5aec13 Bisecting: 5 revisions left to test after this (roughly 3 steps) [3399dd9f372bf0f4f3fb1fb11eb60ebd7b52d79c] Merge branch 'BPF test_progs tests improvement' testing commit 3399dd9f372bf0f4f3fb1fb11eb60ebd7b52d79c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a2066e1869784c7856c2f3f40d20259a407c87d8cbac6e99fe34c6265b73c653 all runs: OK # git bisect good 3399dd9f372bf0f4f3fb1fb11eb60ebd7b52d79c Bisecting: 2 revisions left to test after this (roughly 2 steps) [24592ad1ab18416a850f03d46d07ae483f808895] libbpf: Support batch_size option to bpf_prog_test_run testing commit 24592ad1ab18416a850f03d46d07ae483f808895 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 451b896916c799353e51cb2d45afba63c07ee1f5bf73acafacde1bc83174a670 all runs: crashed: BUG: missing reserved tailroom # git bisect bad 24592ad1ab18416a850f03d46d07ae483f808895 Bisecting: 0 revisions left to test after this (roughly 1 step) [1a7551f15097dff30cf0853117b2f8077bb84f0a] Documentation/bpf: Add documentation for BPF_PROG_RUN testing commit 1a7551f15097dff30cf0853117b2f8077bb84f0a compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 96b82b795e253729e4ec6cb196b7b208a79f41e1033edd27c2a3583c48d9f0b3 all runs: crashed: BUG: missing reserved tailroom # git bisect bad 1a7551f15097dff30cf0853117b2f8077bb84f0a Bisecting: 0 revisions left to test after this (roughly 0 steps) [b530e9e1063ed2b817eae7eec6ed2daa8be11608] bpf: Add "live packet" mode for XDP in BPF_PROG_RUN testing commit b530e9e1063ed2b817eae7eec6ed2daa8be11608 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 57cbf4ffba33d5b0f38afa29150e7573c321ef3df084670e3b328a4465b42055 all runs: crashed: BUG: missing reserved tailroom # git bisect bad b530e9e1063ed2b817eae7eec6ed2daa8be11608 b530e9e1063ed2b817eae7eec6ed2daa8be11608 is the first bad commit commit b530e9e1063ed2b817eae7eec6ed2daa8be11608 Author: Toke Høiland-Jørgensen Date: Wed Mar 9 11:53:42 2022 +0100 bpf: Add "live packet" mode for XDP in BPF_PROG_RUN This adds support for running XDP programs through BPF_PROG_RUN in a mode that enables live packet processing of the resulting frames. Previous uses of BPF_PROG_RUN for XDP returned the XDP program return code and the modified packet data to userspace, which is useful for unit testing of XDP programs. The existing BPF_PROG_RUN for XDP allows userspace to set the ingress ifindex and RXQ number as part of the context object being passed to the kernel. This patch reuses that code, but adds a new mode with different semantics, which can be selected with the new BPF_F_TEST_XDP_LIVE_FRAMES flag. When running BPF_PROG_RUN in this mode, the XDP program return codes will be honoured: returning XDP_PASS will result in the frame being injected into the networking stack as if it came from the selected networking interface, while returning XDP_TX and XDP_REDIRECT will result in the frame being transmitted out that interface. XDP_TX is translated into an XDP_REDIRECT operation to the same interface, since the real XDP_TX action is only possible from within the network drivers themselves, not from the process context where BPF_PROG_RUN is executed. Internally, this new mode of operation creates a page pool instance while setting up the test run, and feeds pages from that into the XDP program. The setup cost of this is amortised over the number of repetitions specified by userspace. To support the performance testing use case, we further optimise the setup step so that all pages in the pool are pre-initialised with the packet data, and pre-computed context and xdp_frame objects stored at the start of each page. This makes it possible to entirely avoid touching the page content on each XDP program invocation, and enables sending up to 9 Mpps/core on my test box. Because the data pages are recycled by the page pool, and the test runner doesn't re-initialise them for each run, subsequent invocations of the XDP program will see the packet data in the state it was after the last time it ran on that particular page. This means that an XDP program that modifies the packet before redirecting it has to be careful about which assumptions it makes about the packet content, but that is only an issue for the most naively written programs. Enabling the new flag is only allowed when not setting ctx_out and data_out in the test specification, since using it means frames will be redirected somewhere else, so they can't be returned. Signed-off-by: Toke Høiland-Jørgensen Signed-off-by: Alexei Starovoitov Acked-by: Martin KaFai Lau Link: https://lore.kernel.org/bpf/20220309105346.100053-2-toke@redhat.com include/uapi/linux/bpf.h | 3 + kernel/bpf/Kconfig | 1 + kernel/bpf/syscall.c | 2 +- net/bpf/test_run.c | 334 +++++++++++++++++++++++++++++++++++++++-- tools/include/uapi/linux/bpf.h | 3 + 5 files changed, 328 insertions(+), 15 deletions(-) culprit signature: 57cbf4ffba33d5b0f38afa29150e7573c321ef3df084670e3b328a4465b42055 parent signature: a2066e1869784c7856c2f3f40d20259a407c87d8cbac6e99fe34c6265b73c653 revisions tested: 23, total time: 4h9m8.715633454s (build: 3h7m1.953212195s, test: 1h0m9.110018875s) first bad commit: b530e9e1063ed2b817eae7eec6ed2daa8be11608 bpf: Add "live packet" mode for XDP in BPF_PROG_RUN recipients (to): ["ast@kernel.org" "kafai@fb.com" "toke@redhat.com"] recipients (cc): [] crash: BUG: missing reserved tailroom ------------[ cut here ]------------ XDP_WARN: xdp_update_frame_from_buff(line:274): Driver BUG: missing reserved tailroom WARNING: CPU: 0 PID: 4053 at net/core/xdp.c:599 xdp_warn+0xf/0x20 net/core/xdp.c:599 Modules linked in: CPU: 0 PID: 4053 Comm: syz-executor387 Not tainted 5.17.0-rc6-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:xdp_warn+0xf/0x20 net/core/xdp.c:599 Code: 03 38 d0 7c 04 84 d2 75 09 83 7b 0c 01 5b 0f 94 c0 c3 e8 64 19 09 fb eb f0 66 90 48 89 f9 48 c7 c7 e0 bf e2 89 e8 de cf 9f 01 <0f> 0b c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 b8 00 00 00 RSP: 0018:ffffc900041df708 EFLAGS: 00010286 RAX: 0000000000000000 RBX: ffff88807e150198 RCX: 0000000000000000 RDX: 0000000000000201 RSI: 0000000000000008 RDI: fffff5200083bed3 RBP: ffff88807e150000 R08: 0000000000000000 R09: ffffc900041df37f R10: fffff5200083be6f R11: 0000000000000001 R12: ffff88807e150058 R13: 0000000000000d00 R14: 0000000000000000 R15: 0000000000000000 FS: 0000555555dc3300(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000020001000 CR3: 0000000074372000 CR4: 00000000003506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: xdp_update_frame_from_buff include/net/xdp.h:274 [inline] xdp_update_frame_from_buff include/net/xdp.h:260 [inline] xdp_test_run_init_page+0x3ae/0x5f0 net/bpf/test_run.c:143 page_pool_set_pp_info net/core/page_pool.c:268 [inline] __page_pool_alloc_pages_slow+0x1db/0xcb0 net/core/page_pool.c:339 page_pool_dev_alloc_pages include/net/page_pool.h:197 [inline] xdp_test_run_batch net/bpf/test_run.c:280 [inline] bpf_test_run_xdp_live+0x4c4/0x1710 net/bpf/test_run.c:363 bpf_prog_test_run_xdp+0x721/0x11f0 net/bpf/test_run.c:1317 bpf_prog_test_run kernel/bpf/syscall.c:3363 [inline] __sys_bpf+0x1074/0x4430 kernel/bpf/syscall.c:4665 __do_sys_bpf kernel/bpf/syscall.c:4751 [inline] __se_sys_bpf kernel/bpf/syscall.c:4749 [inline] __x64_sys_bpf+0x70/0xb0 kernel/bpf/syscall.c:4749 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7f5232215d29 Code: 28 c3 e8 2a 14 00 00 66 2e 0f 1f 84 00 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 c0 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffec874bb38 EFLAGS: 00000246 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f5232215d29 RDX: 0000000000000048 RSI: 0000000020000000 RDI: 000000000000000a RBP: 00007f52321d9ed0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00007f52321d9f60 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000