bisecting fixing commit since 3bd837bfe431839a378e9d421af05b2e22a6d329 building syzkaller on 4ec4ea48904fe8b1ddfe85e84ea117b9dfdc90f2 testing commit 3bd837bfe431839a378e9d421af05b2e22a6d329 with gcc (GCC) 8.1.0 kernel signature: 8705e714a5235a2a17024fb3e9dd5f9d6e2fde85 run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: crashed: WARNING in bpf_jit_free run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK testing current HEAD 312017a460d5ea31d646e7148e400e13db799ddc testing commit 312017a460d5ea31d646e7148e400e13db799ddc with gcc (GCC) 8.1.0 kernel signature: 257eade3670a4fb95be80ecc58a07e9b2d0beb3e all runs: OK # git bisect start 312017a460d5ea31d646e7148e400e13db799ddc 3bd837bfe431839a378e9d421af05b2e22a6d329 Bisecting: 1994 revisions left to test after this (roughly 11 steps) [db1e664e2d8075536741caa3f1d99920cf7190c1] mips: Loongson: Fix the link time qualifier of 'serial_exit()' testing commit db1e664e2d8075536741caa3f1d99920cf7190c1 with gcc (GCC) 8.1.0 kernel signature: 6c12bc66ad79e6536e7a4d6d92f2507169ace079 all runs: OK # git bisect bad db1e664e2d8075536741caa3f1d99920cf7190c1 Bisecting: 997 revisions left to test after this (roughly 10 steps) [7faef13e6f68d50aa50846cfc79ae499f82becf6] lib: logic_pio: Avoid possible overlap for unregistering regions testing commit 7faef13e6f68d50aa50846cfc79ae499f82becf6 with gcc (GCC) 8.1.0 kernel signature: bd354445b5a91def25acae5b7364c260bb85cd65 run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 7faef13e6f68d50aa50846cfc79ae499f82becf6 Bisecting: 498 revisions left to test after this (roughly 9 steps) [8f9dc49928c2fe0e5d4eeab237b76abde1f46e56] leds: leds-lp5562 allow firmware files up to the maximum length testing commit 8f9dc49928c2fe0e5d4eeab237b76abde1f46e56 with gcc (GCC) 8.1.0 kernel signature: abeb006cecb2b218e19e873d37c983a021c46f79 run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: crashed: WARNING in bpf_jit_free run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 8f9dc49928c2fe0e5d4eeab237b76abde1f46e56 Bisecting: 249 revisions left to test after this (roughly 8 steps) [22c788ba7a52881f2458b5571c2e36fb3a3f74f3] hypfs: Fix error number left in struct pointer member testing commit 22c788ba7a52881f2458b5571c2e36fb3a3f74f3 with gcc (GCC) 8.1.0 kernel signature: b63c1ccd3e56266f5fc5f01e647b43cc46b1efca run #0: crashed: WARNING in bpf_jit_free run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 22c788ba7a52881f2458b5571c2e36fb3a3f74f3 Bisecting: 124 revisions left to test after this (roughly 7 steps) [d286a37471b6a72908d1a0214c79fb16e958a28e] riscv: Avoid interrupts being erroneously enabled in handle_exception() testing commit d286a37471b6a72908d1a0214c79fb16e958a28e with gcc (GCC) 8.1.0 kernel signature: 569532f00c115032ce381a89b9ad65ba69ca9b04 all runs: OK # git bisect bad d286a37471b6a72908d1a0214c79fb16e958a28e Bisecting: 62 revisions left to test after this (roughly 6 steps) [561bf9309209e174035eaee0edfc3247ecfdff76] crypto: caam - fix concurrency issue in givencrypt descriptor testing commit 561bf9309209e174035eaee0edfc3247ecfdff76 with gcc (GCC) 8.1.0 kernel signature: 962795ebe6c7c1f8c323a5a48ec179ea02f9e51e all runs: OK # git bisect bad 561bf9309209e174035eaee0edfc3247ecfdff76 Bisecting: 30 revisions left to test after this (roughly 5 steps) [322753c78d1d9d33b0476735cd49f3e6bca5586d] soundwire: Kconfig: fix help format testing commit 322753c78d1d9d33b0476735cd49f3e6bca5586d with gcc (GCC) 8.1.0 kernel signature: 8b895e731b460ccd5bd71e5affb372fe470ff01b all runs: OK # git bisect bad 322753c78d1d9d33b0476735cd49f3e6bca5586d Bisecting: 15 revisions left to test after this (roughly 4 steps) [6f8564edf8c193cfc59dcb68dd7d7297f9a72f39] ipv6: Handle missing host route in __ipv6_ifa_notify testing commit 6f8564edf8c193cfc59dcb68dd7d7297f9a72f39 with gcc (GCC) 8.1.0 kernel signature: 6474180bfb98b595ae048f81438e776a5f553ca7 all runs: OK # git bisect bad 6f8564edf8c193cfc59dcb68dd7d7297f9a72f39 Bisecting: 7 revisions left to test after this (roughly 3 steps) [f91a9c6591c0bf6ef72220ad1041331aadf1d2a2] arm: properly account for stack randomization and stack guard gap testing commit f91a9c6591c0bf6ef72220ad1041331aadf1d2a2 with gcc (GCC) 8.1.0 kernel signature: dca413d5f649907465b88d68dca1451e202b6ac3 run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: crashed: WARNING in bpf_jit_free run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good f91a9c6591c0bf6ef72220ad1041331aadf1d2a2 Bisecting: 3 revisions left to test after this (roughly 2 steps) [2b83891122921c4698c8229ea22d618dc509af2c] cxgb4:Fix out-of-bounds MSI-X info array access testing commit 2b83891122921c4698c8229ea22d618dc509af2c with gcc (GCC) 8.1.0 kernel signature: 264d4629f834a8cdacbf834377c1fa1dc4c750fb all runs: OK # git bisect bad 2b83891122921c4698c8229ea22d618dc509af2c Bisecting: 1 revision left to test after this (roughly 1 step) [dbb7339cfddf7ae2cc00ce260f197fb954300bae] block: mq-deadline: Fix queue restart handling testing commit dbb7339cfddf7ae2cc00ce260f197fb954300bae with gcc (GCC) 8.1.0 kernel signature: 254fe6023343ed4a9db953fdc116efb75ab9f792 run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: boot failed: can't ssh into the instance # git bisect good dbb7339cfddf7ae2cc00ce260f197fb954300bae Bisecting: 0 revisions left to test after this (roughly 0 steps) [ed568ca736012e87176e481582a6eb031cc5fa5e] bpf: fix use after free in prog symbol exposure testing commit ed568ca736012e87176e481582a6eb031cc5fa5e with gcc (GCC) 8.1.0 kernel signature: a847f30460d3142f2a0087a6b1a79839a8c594f5 all runs: OK # git bisect bad ed568ca736012e87176e481582a6eb031cc5fa5e ed568ca736012e87176e481582a6eb031cc5fa5e is the first bad commit commit ed568ca736012e87176e481582a6eb031cc5fa5e Author: Daniel Borkmann Date: Fri Oct 4 10:40:58 2019 -0700 bpf: fix use after free in prog symbol exposure commit c751798aa224fadc5124b49eeb38fb468c0fa039 upstream. syzkaller managed to trigger the warning in bpf_jit_free() which checks via bpf_prog_kallsyms_verify_off() for potentially unlinked JITed BPF progs in kallsyms, and subsequently trips over GPF when walking kallsyms entries: [...] 8021q: adding VLAN 0 to HW filter on device batadv0 8021q: adding VLAN 0 to HW filter on device batadv0 WARNING: CPU: 0 PID: 9869 at kernel/bpf/core.c:810 bpf_jit_free+0x1e8/0x2a0 Kernel panic - not syncing: panic_on_warn set ... CPU: 0 PID: 9869 Comm: kworker/0:7 Not tainted 5.0.0-rc8+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events bpf_prog_free_deferred Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x113/0x167 lib/dump_stack.c:113 panic+0x212/0x40b kernel/panic.c:214 __warn.cold.8+0x1b/0x38 kernel/panic.c:571 report_bug+0x1a4/0x200 lib/bug.c:186 fixup_bug arch/x86/kernel/traps.c:178 [inline] do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:271 do_invalid_op+0x36/0x40 arch/x86/kernel/traps.c:290 invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:973 RIP: 0010:bpf_jit_free+0x1e8/0x2a0 Code: 02 4c 89 e2 83 e2 07 38 d0 7f 08 84 c0 0f 85 86 00 00 00 48 ba 00 02 00 00 00 00 ad de 0f b6 43 02 49 39 d6 0f 84 5f fe ff ff <0f> 0b e9 58 fe ff ff 48 b8 00 00 00 00 00 fc ff df 4c 89 e2 48 c1 RSP: 0018:ffff888092f67cd8 EFLAGS: 00010202 RAX: 0000000000000007 RBX: ffffc90001947000 RCX: ffffffff816e9d88 RDX: dead000000000200 RSI: 0000000000000008 RDI: ffff88808769f7f0 RBP: ffff888092f67d00 R08: fffffbfff1394059 R09: fffffbfff1394058 R10: fffffbfff1394058 R11: ffffffff89ca02c7 R12: ffffc90001947002 R13: ffffc90001947020 R14: ffffffff881eca80 R15: ffff88808769f7e8 BUG: unable to handle kernel paging request at fffffbfff400d000 #PF error: [normal kernel read fault] PGD 21ffee067 P4D 21ffee067 PUD 21ffed067 PMD 9f942067 PTE 0 Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 PID: 9869 Comm: kworker/0:7 Not tainted 5.0.0-rc8+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events bpf_prog_free_deferred RIP: 0010:bpf_get_prog_addr_region kernel/bpf/core.c:495 [inline] RIP: 0010:bpf_tree_comp kernel/bpf/core.c:558 [inline] RIP: 0010:__lt_find include/linux/rbtree_latch.h:115 [inline] RIP: 0010:latch_tree_find include/linux/rbtree_latch.h:208 [inline] RIP: 0010:bpf_prog_kallsyms_find+0x107/0x2e0 kernel/bpf/core.c:632 Code: 00 f0 ff ff 44 38 c8 7f 08 84 c0 0f 85 fa 00 00 00 41 f6 45 02 01 75 02 0f 0b 48 39 da 0f 82 92 00 00 00 48 89 d8 48 c1 e8 03 <42> 0f b6 04 30 84 c0 74 08 3c 03 0f 8e 45 01 00 00 8b 03 48 c1 e0 [...] Upon further debugging, it turns out that whenever we trigger this issue, the kallsyms removal in bpf_prog_ksym_node_del() was /skipped/ but yet bpf_jit_free() reported that the entry is /in use/. Problem is that symbol exposure via bpf_prog_kallsyms_add() but also perf_event_bpf_event() were done /after/ bpf_prog_new_fd(). Once the fd is exposed to the public, a parallel close request came in right before we attempted to do the bpf_prog_kallsyms_add(). Given at this time the prog reference count is one, we start to rip everything underneath us via bpf_prog_release() -> bpf_prog_put(). The memory is eventually released via deferred free, so we're seeing that bpf_jit_free() has a kallsym entry because we added it from bpf_prog_load() but /after/ bpf_prog_put() from the remote CPU. Therefore, move both notifications /before/ we install the fd. The issue was never seen between bpf_prog_alloc_id() and bpf_prog_new_fd() because upon bpf_prog_get_fd_by_id() we'll take another reference to the BPF prog, so we're still holding the original reference from the bpf_prog_load(). Fixes: 6ee52e2a3fe4 ("perf, bpf: Introduce PERF_RECORD_BPF_EVENT") Fixes: 74451e66d516 ("bpf: make jited programs visible in traces") Reported-by: syzbot+bd3bba6ff3fcea7a6ec6@syzkaller.appspotmail.com Signed-off-by: Daniel Borkmann Cc: Song Liu Signed-off-by: Zubin Mithra Signed-off-by: Sasha Levin kernel/bpf/syscall.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) culprit signature: a847f30460d3142f2a0087a6b1a79839a8c594f5 parent signature: 254fe6023343ed4a9db953fdc116efb75ab9f792 revisions tested: 14, total time: 4h24m44.027339852s (build: 2h0m53.811266467s, test: 2h22m21.861894901s) first good commit: ed568ca736012e87176e481582a6eb031cc5fa5e bpf: fix use after free in prog symbol exposure cc: ["daniel@iogearbox.net" "sashal@kernel.org" "zsm@chromium.org"]