bisecting fixing commit since fa5941f45d7ed070118b7c209b7f2c3a034293bd building syzkaller on 618456b4f4f74528ac6b9d709b1870d0e1d70eb2 testing commit fa5941f45d7ed070118b7c209b7f2c3a034293bd with gcc (GCC) 8.1.0 kernel signature: 40801e1c84b2ae9f49e326d095a3644627400b95 run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: crashed: WARNING in bpf_jit_free run #3: crashed: WARNING in bpf_jit_free run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK testing current HEAD a844dc4c544291470aa69edbe2434b040794e269 testing commit a844dc4c544291470aa69edbe2434b040794e269 with gcc (GCC) 8.1.0 kernel signature: 7075d884f21d3a865cbd44ca0170d813d8a9f482 all runs: OK # git bisect start a844dc4c544291470aa69edbe2434b040794e269 fa5941f45d7ed070118b7c209b7f2c3a034293bd Bisecting: 1838 revisions left to test after this (roughly 11 steps) [0570fc57e3421f6b3196691bfd766a01d481bc27] gpiolib: never report open-drain/source lines as 'input' to user-space testing commit 0570fc57e3421f6b3196691bfd766a01d481bc27 with gcc (GCC) 8.1.0 kernel signature: 770f2ad728fb41f18f943da9f074f39331ab44c2 run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 0570fc57e3421f6b3196691bfd766a01d481bc27 Bisecting: 919 revisions left to test after this (roughly 10 steps) [2e3f0caf9f62735a5d997003bf1e17bedd0014d6] ALSA: bebob: fix to detect configured source of sampling clock for Focusrite Saffire Pro i/o series testing commit 2e3f0caf9f62735a5d997003bf1e17bedd0014d6 with gcc (GCC) 8.1.0 kernel signature: 40117e3f51ebdc089d5cfa1726702de9b1a7b37e all runs: OK # git bisect bad 2e3f0caf9f62735a5d997003bf1e17bedd0014d6 Bisecting: 459 revisions left to test after this (roughly 9 steps) [6682e6f68f1f62965c9a55fdc4e4c1ae4ecf2426] clk: at91: select parent if main oscillator or bypass is enabled testing commit 6682e6f68f1f62965c9a55fdc4e4c1ae4ecf2426 with gcc (GCC) 8.1.0 kernel signature: aaa0925c0a27e690d66ac484e70cedc863383b03 run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: crashed: WARNING in bpf_jit_free run #3: crashed: WARNING in bpf_jit_free run #4: crashed: WARNING in bpf_jit_free run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 6682e6f68f1f62965c9a55fdc4e4c1ae4ecf2426 Bisecting: 229 revisions left to test after this (roughly 8 steps) [8a82aee7bdfd5bfcabdc741b1051dae98b576f9c] arm64: Add MIDR encoding for Arm Cortex-A55 and Cortex-A35 testing commit 8a82aee7bdfd5bfcabdc741b1051dae98b576f9c with gcc (GCC) 8.1.0 kernel signature: e4773a3986ae69d52f1f182b37d3a102b6b5de0e all runs: OK # git bisect bad 8a82aee7bdfd5bfcabdc741b1051dae98b576f9c Bisecting: 114 revisions left to test after this (roughly 7 steps) [88ec6870dbe4edcf480f555430549f512084d84e] xhci: Increase STS_SAVE timeout in xhci_suspend() testing commit 88ec6870dbe4edcf480f555430549f512084d84e with gcc (GCC) 8.1.0 kernel signature: ecc5e89d9c80dfda5be8dd9b7fbbd1c05d831ffb all runs: OK # git bisect bad 88ec6870dbe4edcf480f555430549f512084d84e Bisecting: 57 revisions left to test after this (roughly 6 steps) [61a2c2c94422b4d1a07405d1df9408fa3af2b4c6] crypto: skcipher - Unmap pages after an external error testing commit 61a2c2c94422b4d1a07405d1df9408fa3af2b4c6 with gcc (GCC) 8.1.0 kernel signature: c7e61191d9c25edcecf3e58e7c3b0b6f5b205e8a all runs: OK # git bisect bad 61a2c2c94422b4d1a07405d1df9408fa3af2b4c6 Bisecting: 28 revisions left to test after this (roughly 5 steps) [dca8aabd7198e1aa7210ff2de081befba79d0d41] net: qlogic: Fix memory leak in ql_alloc_large_buffers testing commit dca8aabd7198e1aa7210ff2de081befba79d0d41 with gcc (GCC) 8.1.0 kernel signature: d056873d974615af2ed5721aea3ff853da1417f7 all runs: OK # git bisect bad dca8aabd7198e1aa7210ff2de081befba79d0d41 Bisecting: 13 revisions left to test after this (roughly 4 steps) [4b1e27b3b4659ad2c9e49fc52405d751a38acc81] security: smack: Fix possible null-pointer dereferences in smack_socket_sock_rcv_skb() testing commit 4b1e27b3b4659ad2c9e49fc52405d751a38acc81 with gcc (GCC) 8.1.0 kernel signature: a82843ca21a0bad718d6f3488c03f42e0467d417 run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: crashed: WARNING in bpf_jit_free run #3: crashed: WARNING in bpf_jit_free run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 4b1e27b3b4659ad2c9e49fc52405d751a38acc81 Bisecting: 6 revisions left to test after this (roughly 3 steps) [f892d2f0a9adc8ec1e632ddb3163e5227beb7e52] kmemleak: increase DEBUG_KMEMLEAK_EARLY_LOG_SIZE default to 16K testing commit f892d2f0a9adc8ec1e632ddb3163e5227beb7e52 with gcc (GCC) 8.1.0 kernel signature: 6ba8ed81b04060c6b049d1ab565da7149bbd4e93 run #0: crashed: WARNING in bpf_jit_free run #1: crashed: WARNING in bpf_jit_free run #2: crashed: WARNING in bpf_jit_free run #3: crashed: WARNING in bpf_jit_free run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good f892d2f0a9adc8ec1e632ddb3163e5227beb7e52 Bisecting: 3 revisions left to test after this (roughly 2 steps) [881d4609006d2dc59a9fc04739ce042fcf7888c9] erspan: remove the incorrect mtu limit for erspan testing commit 881d4609006d2dc59a9fc04739ce042fcf7888c9 with gcc (GCC) 8.1.0 kernel signature: 8a4db4566cbb7279fe99ae426f077280b3d2fe76 all runs: OK # git bisect bad 881d4609006d2dc59a9fc04739ce042fcf7888c9 Bisecting: 0 revisions left to test after this (roughly 1 step) [382aa3a910a6ea6073b22432b688ce09e7116705] cxgb4:Fix out-of-bounds MSI-X info array access testing commit 382aa3a910a6ea6073b22432b688ce09e7116705 with gcc (GCC) 8.1.0 kernel signature: 2a282b8497d97eb31f35007dbf9f394b339e9dcf all runs: OK # git bisect bad 382aa3a910a6ea6073b22432b688ce09e7116705 Bisecting: 0 revisions left to test after this (roughly 0 steps) [47569360be87709e690e9261df738080a2f740d2] bpf: fix use after free in prog symbol exposure testing commit 47569360be87709e690e9261df738080a2f740d2 with gcc (GCC) 8.1.0 kernel signature: 3cb30370e8ad4872edfe252b4330ad5753b13daf all runs: OK # git bisect bad 47569360be87709e690e9261df738080a2f740d2 47569360be87709e690e9261df738080a2f740d2 is the first bad commit commit 47569360be87709e690e9261df738080a2f740d2 Author: Daniel Borkmann Date: Fri Oct 4 10:41:12 2019 -0700 bpf: fix use after free in prog symbol exposure commit c751798aa224fadc5124b49eeb38fb468c0fa039 upstream. syzkaller managed to trigger the warning in bpf_jit_free() which checks via bpf_prog_kallsyms_verify_off() for potentially unlinked JITed BPF progs in kallsyms, and subsequently trips over GPF when walking kallsyms entries: [...] 8021q: adding VLAN 0 to HW filter on device batadv0 8021q: adding VLAN 0 to HW filter on device batadv0 WARNING: CPU: 0 PID: 9869 at kernel/bpf/core.c:810 bpf_jit_free+0x1e8/0x2a0 Kernel panic - not syncing: panic_on_warn set ... CPU: 0 PID: 9869 Comm: kworker/0:7 Not tainted 5.0.0-rc8+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events bpf_prog_free_deferred Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x113/0x167 lib/dump_stack.c:113 panic+0x212/0x40b kernel/panic.c:214 __warn.cold.8+0x1b/0x38 kernel/panic.c:571 report_bug+0x1a4/0x200 lib/bug.c:186 fixup_bug arch/x86/kernel/traps.c:178 [inline] do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:271 do_invalid_op+0x36/0x40 arch/x86/kernel/traps.c:290 invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:973 RIP: 0010:bpf_jit_free+0x1e8/0x2a0 Code: 02 4c 89 e2 83 e2 07 38 d0 7f 08 84 c0 0f 85 86 00 00 00 48 ba 00 02 00 00 00 00 ad de 0f b6 43 02 49 39 d6 0f 84 5f fe ff ff <0f> 0b e9 58 fe ff ff 48 b8 00 00 00 00 00 fc ff df 4c 89 e2 48 c1 RSP: 0018:ffff888092f67cd8 EFLAGS: 00010202 RAX: 0000000000000007 RBX: ffffc90001947000 RCX: ffffffff816e9d88 RDX: dead000000000200 RSI: 0000000000000008 RDI: ffff88808769f7f0 RBP: ffff888092f67d00 R08: fffffbfff1394059 R09: fffffbfff1394058 R10: fffffbfff1394058 R11: ffffffff89ca02c7 R12: ffffc90001947002 R13: ffffc90001947020 R14: ffffffff881eca80 R15: ffff88808769f7e8 BUG: unable to handle kernel paging request at fffffbfff400d000 #PF error: [normal kernel read fault] PGD 21ffee067 P4D 21ffee067 PUD 21ffed067 PMD 9f942067 PTE 0 Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 PID: 9869 Comm: kworker/0:7 Not tainted 5.0.0-rc8+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events bpf_prog_free_deferred RIP: 0010:bpf_get_prog_addr_region kernel/bpf/core.c:495 [inline] RIP: 0010:bpf_tree_comp kernel/bpf/core.c:558 [inline] RIP: 0010:__lt_find include/linux/rbtree_latch.h:115 [inline] RIP: 0010:latch_tree_find include/linux/rbtree_latch.h:208 [inline] RIP: 0010:bpf_prog_kallsyms_find+0x107/0x2e0 kernel/bpf/core.c:632 Code: 00 f0 ff ff 44 38 c8 7f 08 84 c0 0f 85 fa 00 00 00 41 f6 45 02 01 75 02 0f 0b 48 39 da 0f 82 92 00 00 00 48 89 d8 48 c1 e8 03 <42> 0f b6 04 30 84 c0 74 08 3c 03 0f 8e 45 01 00 00 8b 03 48 c1 e0 [...] Upon further debugging, it turns out that whenever we trigger this issue, the kallsyms removal in bpf_prog_ksym_node_del() was /skipped/ but yet bpf_jit_free() reported that the entry is /in use/. Problem is that symbol exposure via bpf_prog_kallsyms_add() but also perf_event_bpf_event() were done /after/ bpf_prog_new_fd(). Once the fd is exposed to the public, a parallel close request came in right before we attempted to do the bpf_prog_kallsyms_add(). Given at this time the prog reference count is one, we start to rip everything underneath us via bpf_prog_release() -> bpf_prog_put(). The memory is eventually released via deferred free, so we're seeing that bpf_jit_free() has a kallsym entry because we added it from bpf_prog_load() but /after/ bpf_prog_put() from the remote CPU. Therefore, move both notifications /before/ we install the fd. The issue was never seen between bpf_prog_alloc_id() and bpf_prog_new_fd() because upon bpf_prog_get_fd_by_id() we'll take another reference to the BPF prog, so we're still holding the original reference from the bpf_prog_load(). Fixes: 6ee52e2a3fe4 ("perf, bpf: Introduce PERF_RECORD_BPF_EVENT") Fixes: 74451e66d516 ("bpf: make jited programs visible in traces") Reported-by: syzbot+bd3bba6ff3fcea7a6ec6@syzkaller.appspotmail.com Signed-off-by: Daniel Borkmann Cc: Song Liu Signed-off-by: Zubin Mithra Signed-off-by: Sasha Levin kernel/bpf/syscall.c | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) culprit signature: 3cb30370e8ad4872edfe252b4330ad5753b13daf parent signature: 6ba8ed81b04060c6b049d1ab565da7149bbd4e93 revisions tested: 14, total time: 4h13m46.761887039s (build: 1h51m48.162255914s, test: 2h20m38.510259291s) first good commit: 47569360be87709e690e9261df738080a2f740d2 bpf: fix use after free in prog symbol exposure cc: ["daniel@iogearbox.net" "sashal@kernel.org" "zsm@chromium.org"]