bisecting fixing commit since 6cad9d0cf87b95b10f3f4d7826c2c15e45e2a277 building syzkaller on 28ac6e6496673327d3319bab81c57a0f7366fb45 testing commit 6cad9d0cf87b95b10f3f4d7826c2c15e45e2a277 with gcc (GCC) 8.1.0 kernel signature: 97d928ac4668885b68b74dbd29c5b007e8b2dbdd run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: crashed: WARNING in bpf_jit_free run #3: crashed: WARNING in bpf_jit_free run #4: crashed: WARNING in bpf_jit_free run #5: crashed: WARNING in bpf_jit_free run #6: crashed: WARNING in bpf_jit_free run #7: crashed: WARNING in bpf_jit_free run #8: crashed: WARNING in bpf_jit_free run #9: OK testing current HEAD 312017a460d5ea31d646e7148e400e13db799ddc testing commit 312017a460d5ea31d646e7148e400e13db799ddc with gcc (GCC) 8.1.0 kernel signature: 4e9050ac640190fe13a46abc01a8ad5f7c1bb4fd all runs: OK # git bisect start 312017a460d5ea31d646e7148e400e13db799ddc 6cad9d0cf87b95b10f3f4d7826c2c15e45e2a277 Bisecting: 1156 revisions left to test after this (roughly 10 steps) [6fce50c100c85f1c05f463b2ee0b83be1290f4cb] f2fs: fix remount problem of option io_bits testing commit 6fce50c100c85f1c05f463b2ee0b83be1290f4cb with gcc (GCC) 8.1.0 kernel signature: 445891eec0f0a4a58d38c264812f21c52279dd8e all runs: OK # git bisect bad 6fce50c100c85f1c05f463b2ee0b83be1290f4cb Bisecting: 578 revisions left to test after this (roughly 9 steps) [18e7fae372a10334d34c7ad7a28060a14c10f17e] irqchip/gic-v3-its: Use the exact ITSList for VMOVP testing commit 18e7fae372a10334d34c7ad7a28060a14c10f17e with gcc (GCC) 8.1.0 kernel signature: a24e0cee89eecf681deeea1700194bbe0a3dd173 all runs: OK # git bisect bad 18e7fae372a10334d34c7ad7a28060a14c10f17e Bisecting: 288 revisions left to test after this (roughly 8 steps) [a73306414fcdd9b56af117a330b98ea63ec46d61] ACPICA: ACPI 6.3: PPTT add additional fields in Processor Structure Flags testing commit a73306414fcdd9b56af117a330b98ea63ec46d61 with gcc (GCC) 8.1.0 kernel signature: 49d760a919850d5eacd4b4957085ac8daf112240 all runs: OK # git bisect bad a73306414fcdd9b56af117a330b98ea63ec46d61 Bisecting: 144 revisions left to test after this (roughly 7 steps) [0e45633f49ef5c0b0fcc1a8577c837ed53f374a7] drm/omap: fix max fclk divider for omap36xx testing commit 0e45633f49ef5c0b0fcc1a8577c837ed53f374a7 with gcc (GCC) 8.1.0 kernel signature: 1f22d34d125f5cc05a140d24e69b7c0952704437 all runs: OK # git bisect bad 0e45633f49ef5c0b0fcc1a8577c837ed53f374a7 Bisecting: 71 revisions left to test after this (roughly 6 steps) [52132ff52cadb32edf063c1b4d8a7a6c00d85e4d] ocfs2: wait for recovering done after direct unlock request testing commit 52132ff52cadb32edf063c1b4d8a7a6c00d85e4d with gcc (GCC) 8.1.0 kernel signature: b90abbaf9fc1b6dece27c353865d16f94f16b546 run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: crashed: WARNING in bpf_jit_free run #3: crashed: BUG: unable to handle kernel paging request in bpf_prog_kallsyms_add run #4: crashed: WARNING in bpf_jit_free run #5: crashed: WARNING in bpf_jit_free run #6: crashed: WARNING in bpf_jit_free run #7: crashed: WARNING in bpf_jit_free run #8: crashed: WARNING in bpf_jit_free run #9: OK # git bisect good 52132ff52cadb32edf063c1b4d8a7a6c00d85e4d Bisecting: 35 revisions left to test after this (roughly 5 steps) [8b41a30f91dbc4dfade46e4cc161cb49561f606f] s390/process: avoid potential reading of freed stack testing commit 8b41a30f91dbc4dfade46e4cc161cb49561f606f with gcc (GCC) 8.1.0 kernel signature: 05edabdea23daf3ee512672566dd9acf45b66a45 all runs: OK # git bisect bad 8b41a30f91dbc4dfade46e4cc161cb49561f606f Bisecting: 17 revisions left to test after this (roughly 4 steps) [76b552775d601d8ec13d78b2e2df07aec34b0264] rxrpc: Fix rxrpc_recvmsg tracepoint testing commit 76b552775d601d8ec13d78b2e2df07aec34b0264 with gcc (GCC) 8.1.0 kernel signature: 19323ac0a4afab41ee845e0248f16db109df74f4 all runs: OK # git bisect bad 76b552775d601d8ec13d78b2e2df07aec34b0264 Bisecting: 8 revisions left to test after this (roughly 3 steps) [7f30c44b7ca43e6960f22e7a1efec30fbfed6bea] erspan: remove the incorrect mtu limit for erspan testing commit 7f30c44b7ca43e6960f22e7a1efec30fbfed6bea with gcc (GCC) 8.1.0 kernel signature: 86ca71bc267ec425187517be47eb16fda0fa21da all runs: OK # git bisect bad 7f30c44b7ca43e6960f22e7a1efec30fbfed6bea Bisecting: 4 revisions left to test after this (roughly 2 steps) [f91a9c6591c0bf6ef72220ad1041331aadf1d2a2] arm: properly account for stack randomization and stack guard gap testing commit f91a9c6591c0bf6ef72220ad1041331aadf1d2a2 with gcc (GCC) 8.1.0 kernel signature: d9d0a5d53f95cbcbeb7074d0995427d3c2060fab run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: crashed: WARNING in bpf_jit_free run #3: crashed: WARNING in bpf_jit_free run #4: crashed: WARNING in bpf_jit_free run #5: crashed: WARNING in bpf_jit_free run #6: crashed: WARNING in bpf_jit_free run #7: crashed: WARNING in bpf_jit_free run #8: crashed: WARNING in bpf_jit_free run #9: OK # git bisect good f91a9c6591c0bf6ef72220ad1041331aadf1d2a2 Bisecting: 2 revisions left to test after this (roughly 1 step) [dbb7339cfddf7ae2cc00ce260f197fb954300bae] block: mq-deadline: Fix queue restart handling testing commit dbb7339cfddf7ae2cc00ce260f197fb954300bae with gcc (GCC) 8.1.0 kernel signature: 1672596ef39b0e2acbd448baf3503a0b52aedb77 all runs: crashed: WARNING in bpf_jit_free # git bisect good dbb7339cfddf7ae2cc00ce260f197fb954300bae Bisecting: 0 revisions left to test after this (roughly 1 step) [2b83891122921c4698c8229ea22d618dc509af2c] cxgb4:Fix out-of-bounds MSI-X info array access testing commit 2b83891122921c4698c8229ea22d618dc509af2c with gcc (GCC) 8.1.0 kernel signature: 51f95d3d67660bc8a7a1570419a06fc1e3dc82bc all runs: OK # git bisect bad 2b83891122921c4698c8229ea22d618dc509af2c Bisecting: 0 revisions left to test after this (roughly 0 steps) [ed568ca736012e87176e481582a6eb031cc5fa5e] bpf: fix use after free in prog symbol exposure testing commit ed568ca736012e87176e481582a6eb031cc5fa5e with gcc (GCC) 8.1.0 kernel signature: 712c5fd975aafdb81e98780250ec919772967648 all runs: OK # git bisect bad ed568ca736012e87176e481582a6eb031cc5fa5e ed568ca736012e87176e481582a6eb031cc5fa5e is the first bad commit commit ed568ca736012e87176e481582a6eb031cc5fa5e Author: Daniel Borkmann Date: Fri Oct 4 10:40:58 2019 -0700 bpf: fix use after free in prog symbol exposure commit c751798aa224fadc5124b49eeb38fb468c0fa039 upstream. syzkaller managed to trigger the warning in bpf_jit_free() which checks via bpf_prog_kallsyms_verify_off() for potentially unlinked JITed BPF progs in kallsyms, and subsequently trips over GPF when walking kallsyms entries: [...] 8021q: adding VLAN 0 to HW filter on device batadv0 8021q: adding VLAN 0 to HW filter on device batadv0 WARNING: CPU: 0 PID: 9869 at kernel/bpf/core.c:810 bpf_jit_free+0x1e8/0x2a0 Kernel panic - not syncing: panic_on_warn set ... CPU: 0 PID: 9869 Comm: kworker/0:7 Not tainted 5.0.0-rc8+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events bpf_prog_free_deferred Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x113/0x167 lib/dump_stack.c:113 panic+0x212/0x40b kernel/panic.c:214 __warn.cold.8+0x1b/0x38 kernel/panic.c:571 report_bug+0x1a4/0x200 lib/bug.c:186 fixup_bug arch/x86/kernel/traps.c:178 [inline] do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:271 do_invalid_op+0x36/0x40 arch/x86/kernel/traps.c:290 invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:973 RIP: 0010:bpf_jit_free+0x1e8/0x2a0 Code: 02 4c 89 e2 83 e2 07 38 d0 7f 08 84 c0 0f 85 86 00 00 00 48 ba 00 02 00 00 00 00 ad de 0f b6 43 02 49 39 d6 0f 84 5f fe ff ff <0f> 0b e9 58 fe ff ff 48 b8 00 00 00 00 00 fc ff df 4c 89 e2 48 c1 RSP: 0018:ffff888092f67cd8 EFLAGS: 00010202 RAX: 0000000000000007 RBX: ffffc90001947000 RCX: ffffffff816e9d88 RDX: dead000000000200 RSI: 0000000000000008 RDI: ffff88808769f7f0 RBP: ffff888092f67d00 R08: fffffbfff1394059 R09: fffffbfff1394058 R10: fffffbfff1394058 R11: ffffffff89ca02c7 R12: ffffc90001947002 R13: ffffc90001947020 R14: ffffffff881eca80 R15: ffff88808769f7e8 BUG: unable to handle kernel paging request at fffffbfff400d000 #PF error: [normal kernel read fault] PGD 21ffee067 P4D 21ffee067 PUD 21ffed067 PMD 9f942067 PTE 0 Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 PID: 9869 Comm: kworker/0:7 Not tainted 5.0.0-rc8+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events bpf_prog_free_deferred RIP: 0010:bpf_get_prog_addr_region kernel/bpf/core.c:495 [inline] RIP: 0010:bpf_tree_comp kernel/bpf/core.c:558 [inline] RIP: 0010:__lt_find include/linux/rbtree_latch.h:115 [inline] RIP: 0010:latch_tree_find include/linux/rbtree_latch.h:208 [inline] RIP: 0010:bpf_prog_kallsyms_find+0x107/0x2e0 kernel/bpf/core.c:632 Code: 00 f0 ff ff 44 38 c8 7f 08 84 c0 0f 85 fa 00 00 00 41 f6 45 02 01 75 02 0f 0b 48 39 da 0f 82 92 00 00 00 48 89 d8 48 c1 e8 03 <42> 0f b6 04 30 84 c0 74 08 3c 03 0f 8e 45 01 00 00 8b 03 48 c1 e0 [...] Upon further debugging, it turns out that whenever we trigger this issue, the kallsyms removal in bpf_prog_ksym_node_del() was /skipped/ but yet bpf_jit_free() reported that the entry is /in use/. Problem is that symbol exposure via bpf_prog_kallsyms_add() but also perf_event_bpf_event() were done /after/ bpf_prog_new_fd(). Once the fd is exposed to the public, a parallel close request came in right before we attempted to do the bpf_prog_kallsyms_add(). Given at this time the prog reference count is one, we start to rip everything underneath us via bpf_prog_release() -> bpf_prog_put(). The memory is eventually released via deferred free, so we're seeing that bpf_jit_free() has a kallsym entry because we added it from bpf_prog_load() but /after/ bpf_prog_put() from the remote CPU. Therefore, move both notifications /before/ we install the fd. The issue was never seen between bpf_prog_alloc_id() and bpf_prog_new_fd() because upon bpf_prog_get_fd_by_id() we'll take another reference to the BPF prog, so we're still holding the original reference from the bpf_prog_load(). Fixes: 6ee52e2a3fe4 ("perf, bpf: Introduce PERF_RECORD_BPF_EVENT") Fixes: 74451e66d516 ("bpf: make jited programs visible in traces") Reported-by: syzbot+bd3bba6ff3fcea7a6ec6@syzkaller.appspotmail.com Signed-off-by: Daniel Borkmann Cc: Song Liu Signed-off-by: Zubin Mithra Signed-off-by: Sasha Levin kernel/bpf/syscall.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) culprit signature: 712c5fd975aafdb81e98780250ec919772967648 parent signature: 1672596ef39b0e2acbd448baf3503a0b52aedb77 revisions tested: 14, total time: 4h18m9.86281476s (build: 1h58m34.161420173s, test: 2h18m9.483295982s) first good commit: ed568ca736012e87176e481582a6eb031cc5fa5e bpf: fix use after free in prog symbol exposure cc: ["daniel@iogearbox.net" "sashal@kernel.org" "zsm@chromium.org"]