bisecting cause commit starting from 156c05917e0920ef5643eb54c0ea71aae5d60c3d building syzkaller on a547defcdc2fffb71a997c3bff70cecbc1572c3d testing commit 156c05917e0920ef5643eb54c0ea71aae5d60c3d with gcc (GCC) 8.1.0 run #0: crashed: KASAN: slab-out-of-bounds Read in class_equal run #1: crashed: KASAN: use-after-free Read in class_equal run #2: crashed: kernel panic: corrupted stack end in corrupted run #3: crashed: KASAN: slab-out-of-bounds Read in class_equal run #4: crashed: KASAN: use-after-free Read in class_equal run #5: crashed: KASAN: use-after-free Read in class_equal run #6: crashed: KASAN: use-after-free Read in sk_psock_unlink run #7: crashed: KASAN: use-after-free Read in class_equal run #8: crashed: KASAN: slab-out-of-bounds Read in class_equal run #9: crashed: KASAN: slab-out-of-bounds Read in class_equal testing release v5.1 testing commit e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd with gcc (GCC) 8.1.0 all runs: crashed: WARNING: ODEBUG bug in del_timer testing release v5.0 testing commit 1c163f4c7b3f621efff9b28a47abb36f7378d783 with gcc (GCC) 8.1.0 all runs: crashed: WARNING in strp_done testing release v4.20 testing commit 8fe28cb58bcb235034b64cbbb7550a8a43fd88be with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in class_equal run #1: crashed: unexpected kernel reboot run #2: crashed: KASAN: use-after-free Read in class_equal run #3: crashed: kernel panic: corrupted stack end in pgtable_bad run #4: crashed: KASAN: use-after-free Read in class_equal run #5: crashed: KASAN: slab-out-of-bounds Read in class_equal run #6: crashed: KASAN: slab-out-of-bounds Read in class_equal run #7: crashed: KASAN: slab-out-of-bounds Read in class_equal run #8: crashed: kernel panic: corrupted stack end in corrupted run #9: crashed: KASAN: use-after-free Read in class_equal testing release v4.19 testing commit 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in psock_map_pop run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: crashed: KASAN: use-after-free Read in psock_map_pop run #7: OK run #8: crashed: KASAN: use-after-free Read in psock_map_pop run #9: OK testing release v4.18 testing commit 94710cac0ef4ee177a63b5227664b38c95bbf703 with gcc (GCC) 8.1.0 all runs: OK # git bisect start v4.19 v4.18 Bisecting: 7596 revisions left to test after this (roughly 13 steps) [db06f826ec12bf0701ea7fc0a3c0aa00b84417c8] Merge tag 'clk-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux testing commit db06f826ec12bf0701ea7fc0a3c0aa00b84417c8 with gcc (GCC) 8.1.0 all runs: OK # git bisect good db06f826ec12bf0701ea7fc0a3c0aa00b84417c8 Bisecting: 3768 revisions left to test after this (roughly 12 steps) [cd9b44f90763c3367e8dd0601849ffb028e8ba52] Merge branch 'akpm' (patches from Andrew) testing commit cd9b44f90763c3367e8dd0601849ffb028e8ba52 with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in psock_map_pop run #1: crashed: KASAN: use-after-free Read in psock_map_pop run #2: crashed: KASAN: use-after-free Read in psock_map_pop run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad cd9b44f90763c3367e8dd0601849ffb028e8ba52 Bisecting: 2297 revisions left to test after this (roughly 11 steps) [336722eb9d9732c5a497fb6299bf38cde413592b] Merge tag 'tty-4.19-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty testing commit 336722eb9d9732c5a497fb6299bf38cde413592b with gcc (GCC) 8.1.0 all runs: OK # git bisect good 336722eb9d9732c5a497fb6299bf38cde413592b Bisecting: 1122 revisions left to test after this (roughly 10 steps) [d5acba26bfa097a618be425522b1ec4269d3edaf] Merge tag 'char-misc-4.19-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc testing commit d5acba26bfa097a618be425522b1ec4269d3edaf with gcc (GCC) 8.1.0 all runs: OK # git bisect good d5acba26bfa097a618be425522b1ec4269d3edaf Bisecting: 547 revisions left to test after this (roughly 9 steps) [bfebeb16722d93caf7870b63aa7d094b6843479a] Merge tag 'rtc-4.19' of git://git.kernel.org/pub/scm/linux/kernel/git/abelloni/linux testing commit bfebeb16722d93caf7870b63aa7d094b6843479a with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in psock_map_pop run #1: crashed: KASAN: use-after-free Read in psock_map_pop run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad bfebeb16722d93caf7870b63aa7d094b6843479a Bisecting: 300 revisions left to test after this (roughly 8 steps) [1009aa1205c2c5e9101437dcadfa195708d863bf] Merge tag 'riscv-for-linus-4.19-mw0' of git://git.kernel.org/pub/scm/linux/kernel/git/palmer/riscv-linux testing commit 1009aa1205c2c5e9101437dcadfa195708d863bf with gcc (GCC) 8.1.0 all runs: OK # git bisect good 1009aa1205c2c5e9101437dcadfa195708d863bf Bisecting: 158 revisions left to test after this (roughly 7 steps) [8786583db54197b3859311870912f51cb3fca434] Merge tag 'edac_fixes_for_4.19' of git://git.kernel.org/pub/scm/linux/kernel/git/bp/bp testing commit 8786583db54197b3859311870912f51cb3fca434 with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in psock_map_pop run #1: crashed: KASAN: use-after-free Read in psock_map_pop run #2: crashed: KASAN: use-after-free Read in psock_map_pop run #3: crashed: KASAN: use-after-free Read in psock_map_pop run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: crashed: KASAN: use-after-free Read in psock_map_pop # git bisect bad 8786583db54197b3859311870912f51cb3fca434 Bisecting: 70 revisions left to test after this (roughly 6 steps) [e49fcb8b9ef26dfb2d02b173d790e1ef41177121] kvm: nVMX: Fix fault priority for VMX operations testing commit e49fcb8b9ef26dfb2d02b173d790e1ef41177121 with gcc (GCC) 8.1.0 all runs: OK # git bisect good e49fcb8b9ef26dfb2d02b173d790e1ef41177121 Bisecting: 32 revisions left to test after this (roughly 5 steps) [6e3bf9b04f79a009a7ff336ba7353ab565daaec8] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf testing commit 6e3bf9b04f79a009a7ff336ba7353ab565daaec8 with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in psock_map_pop run #1: crashed: KASAN: use-after-free Read in psock_map_pop run #2: crashed: KASAN: use-after-free Read in psock_map_pop run #3: crashed: KASAN: use-after-free Read in psock_map_pop run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 6e3bf9b04f79a009a7ff336ba7353ab565daaec8 Bisecting: 22 revisions left to test after this (roughly 4 steps) [feb9f55c33e5114127238a2c87c069b4f30d1f23] netfilter: nft_dynset: allow dynamic updates of non-anonymous set testing commit feb9f55c33e5114127238a2c87c069b4f30d1f23 with gcc (GCC) 8.1.0 all runs: OK # git bisect good feb9f55c33e5114127238a2c87c069b4f30d1f23 Bisecting: 11 revisions left to test after this (roughly 4 steps) [f6069b9aa9934ede26f41ac0781fce241279ad43] bpf: fix redirect to map under tail calls testing commit f6069b9aa9934ede26f41ac0781fce241279ad43 with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in psock_map_pop run #1: crashed: KASAN: use-after-free Read in psock_map_pop run #2: crashed: KASAN: use-after-free Read in psock_map_pop run #3: crashed: KASAN: use-after-free Read in psock_map_pop run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad f6069b9aa9934ede26f41ac0781fce241279ad43 Bisecting: 5 revisions left to test after this (roughly 3 steps) [90545cdc3f2b2ea700e24335610cd181e73756da] tcp, ulp: fix leftover icsk_ulp_ops preventing sock from reattach testing commit 90545cdc3f2b2ea700e24335610cd181e73756da with gcc (GCC) 8.1.0 all runs: OK # git bisect good 90545cdc3f2b2ea700e24335610cd181e73756da Bisecting: 2 revisions left to test after this (roughly 2 steps) [585f5a6252ee43ec8feeee07387e3fcc7e8bb292] bpf, sockmap: fix sock_map_ctx_update_elem race with exist/noexist testing commit 585f5a6252ee43ec8feeee07387e3fcc7e8bb292 with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in psock_map_pop run #1: crashed: KASAN: use-after-free Read in psock_map_pop run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 585f5a6252ee43ec8feeee07387e3fcc7e8bb292 Bisecting: 0 revisions left to test after this (roughly 1 step) [166ab6f0a0702fdd4d865ad5090bf3094ed83428] bpf, sockmap: fix map elem deletion race with smap_stop_sock testing commit 166ab6f0a0702fdd4d865ad5090bf3094ed83428 with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in psock_map_pop run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 166ab6f0a0702fdd4d865ad5090bf3094ed83428 Bisecting: 0 revisions left to test after this (roughly 0 steps) [d40b0116c94bd8fc2b63aae35ce8e66bb53bba42] bpf, sockmap: fix leakage of smap_psock_map_entry testing commit d40b0116c94bd8fc2b63aae35ce8e66bb53bba42 with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in psock_map_pop run #1: crashed: KASAN: use-after-free Read in psock_map_pop run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad d40b0116c94bd8fc2b63aae35ce8e66bb53bba42 d40b0116c94bd8fc2b63aae35ce8e66bb53bba42 is the first bad commit commit d40b0116c94bd8fc2b63aae35ce8e66bb53bba42 Author: Daniel Borkmann Date: Thu Aug 16 21:49:08 2018 +0200 bpf, sockmap: fix leakage of smap_psock_map_entry While working on sockmap I noticed that we do not always kfree the struct smap_psock_map_entry list elements which track psocks attached to maps. In the case of sock_hash_ctx_update_elem(), these map entries are allocated outside of __sock_map_ctx_update_elem() with their linkage to the socket hash table filled. In the case of sock array, the map entries are allocated inside of __sock_map_ctx_update_elem() and added with their linkage to the psock->maps. Both additions are under psock->maps_lock each. Now, we drop these elements from their psock->maps list in a few occasions: i) in sock array via smap_list_map_remove() when an entry is either deleted from the map from user space, or updated via user space or BPF program where we drop the old socket at that map slot, or the sock array is freed via sock_map_free() and drops all its elements; ii) for sock hash via smap_list_hash_remove() in exactly the same occasions as just described for sock array; iii) in the bpf_tcp_close() where we remove the elements from the list via psock_map_pop() and iterate over them dropping themselves from either sock array or sock hash; and last but not least iv) once again in smap_gc_work() which is a callback for deferring the work once the psock refcount hit zero and thus the socket is being destroyed. Problem is that the only case where we kfree() the list entry is in case iv), which at that point should have an empty list in normal cases. So in cases from i) to iii) we unlink the elements without freeing where they go out of reach from us. Hence fix is to properly kfree() them as well to stop the leakage. Given these are all handled under psock->maps_lock there is no need for deferred RCU freeing. I later also ran with kmemleak detector and it confirmed the finding as well where in the state before the fix the object goes unreferenced while after the patch no kmemleak report related to BPF showed up. [...] unreferenced object 0xffff880378eadae0 (size 64): comm "test_sockmap", pid 2225, jiffies 4294720701 (age 43.504s) hex dump (first 32 bytes): 00 01 00 00 00 00 ad de 00 02 00 00 00 00 ad de ................ 50 4d 75 5d 03 88 ff ff 00 00 00 00 00 00 00 00 PMu]............ backtrace: [<000000005225ac3c>] sock_map_ctx_update_elem.isra.21+0xd8/0x210 [<0000000045dd6d3c>] bpf_sock_map_update+0x29/0x60 [<00000000877723aa>] ___bpf_prog_run+0x1e1f/0x4960 [<000000002ef89e83>] 0xffffffffffffffff unreferenced object 0xffff880378ead240 (size 64): comm "test_sockmap", pid 2225, jiffies 4294720701 (age 43.504s) hex dump (first 32 bytes): 00 01 00 00 00 00 ad de 00 02 00 00 00 00 ad de ................ 00 44 75 5d 03 88 ff ff 00 00 00 00 00 00 00 00 .Du]............ backtrace: [<000000005225ac3c>] sock_map_ctx_update_elem.isra.21+0xd8/0x210 [<0000000030e37a3a>] sock_map_update_elem+0x125/0x240 [<000000002e5ce36e>] map_update_elem+0x4eb/0x7b0 [<00000000db453cc9>] __x64_sys_bpf+0x1f9/0x360 [<0000000000763660>] do_syscall_64+0x9a/0x300 [<00000000422a2bb2>] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [<000000002ef89e83>] 0xffffffffffffffff [...] Fixes: e9db4ef6bf4c ("bpf: sockhash fix omitted bucket lock in sock_close") Fixes: 54fedb42c653 ("bpf: sockmap, fix smap_list_map_remove when psock is in many maps") Fixes: 2f857d04601a ("bpf: sockmap, remove STRPARSER map_flags and add multi-map support") Signed-off-by: Daniel Borkmann Acked-by: John Fastabend Acked-by: Song Liu Signed-off-by: Alexei Starovoitov :040000 040000 14f31df4813db7605156608c242f7f839a92e11f e93c4f003d3a22507523229acec6356a47cb717a M kernel revisions tested: 21, total time: 5h27m43.357675703s (build: 1h51m54.335217953s, test: 3h30m13.235761271s) first bad commit: d40b0116c94bd8fc2b63aae35ce8e66bb53bba42 bpf, sockmap: fix leakage of smap_psock_map_entry cc: ["ast@kernel.org" "daniel@iogearbox.net" "john.fastabend@gmail.com" "songliubraving@fb.com"] crash: KASAN: use-after-free Read in psock_map_pop ================================================================== BUG: KASAN: use-after-free in __lock_acquire+0x3455/0x4950 kernel/locking/lockdep.c:3314 Read of size 8 at addr ffff8800731de708 by task syz-executor.3/14022 CPU: 1 PID: 14022 Comm: syz-executor.3 Not tainted 4.18.0+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x109/0x15a lib/dump_stack.c:113 print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256 kasan_report_error mm/kasan/report.c:354 [inline] kasan_report.cold.9+0x242/0x2fe mm/kasan/report.c:412 __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433 __lock_acquire+0x3455/0x4950 kernel/locking/lockdep.c:3314 lock_acquire+0x173/0x400 kernel/locking/lockdep.c:3924 __raw_spin_lock_bh include/linux/spinlock_api_smp.h:135 [inline] _raw_spin_lock_bh+0x31/0x40 kernel/locking/spinlock.c:168 spin_lock_bh include/linux/spinlock.h:334 [inline] psock_map_pop.isra.26+0x1f/0x1b0 kernel/bpf/sockmap.c:297 bpf_tcp_close+0x49c/0xbc0 kernel/bpf/sockmap.c:374 inet_release+0xd9/0x1c0 net/ipv4/af_inet.c:428 inet6_release+0x46/0x60 net/ipv6/af_inet6.c:457 __sock_release+0xc2/0x230 net/socket.c:579 sock_close+0x10/0x20 net/socket.c:1139 __fput+0x264/0x790 fs/file_table.c:251 ____fput+0x9/0x10 fs/file_table.c:282 task_work_run+0x108/0x180 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:193 [inline] exit_to_usermode_loop+0x1a9/0x200 arch/x86/entry/common.c:166 prepare_exit_to_usermode arch/x86/entry/common.c:197 [inline] syscall_return_slowpath arch/x86/entry/common.c:268 [inline] do_syscall_64+0x407/0x4d0 arch/x86/entry/common.c:293 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x412f61 Code: 75 14 b8 03 00 00 00 0f 05 48 3d 01 f0 ff ff 0f 83 04 1b 00 00 c3 48 83 ec 08 e8 0a fc ff ff 48 89 04 24 b8 03 00 00 00 0f 05 <48> 8b 3c 24 48 89 c2 e8 53 fc ff ff 48 89 d0 48 83 c4 08 48 3d 01 RSP: 002b:00007ffc27122fa0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003 RAX: 0000000000000000 RBX: 0000000000000006 RCX: 0000000000412f61 RDX: 0000000000000000 RSI: 0000000000000081 RDI: 0000000000000005 RBP: 0000000000000000 R08: 0000000000761170 R09: ffffffffffffffff R10: 00007ffc27123070 R11: 0000000000000293 R12: 0000000000761178 R13: 0000000000000005 R14: 0000000000000000 R15: 000000000075bf2c Allocated by task 14038: save_stack+0x43/0xd0 mm/kasan/kasan.c:448 set_track mm/kasan/kasan.c:460 [inline] kasan_kmalloc+0xc7/0xe0 mm/kasan/kasan.c:553 kmem_cache_alloc_node_trace+0x14c/0x770 mm/slab.c:3663 kmalloc_node include/linux/slab.h:551 [inline] kzalloc_node include/linux/slab.h:718 [inline] smap_init_psock kernel/bpf/sockmap.c:1601 [inline] __sock_map_ctx_update_elem.isra.20+0x544/0xd30 kernel/bpf/sockmap.c:1896 sock_map_ctx_update_elem.isra.21+0x134/0x340 kernel/bpf/sockmap.c:1983 sock_map_update_elem+0x199/0x3a0 kernel/bpf/sockmap.c:2086 map_update_elem+0x546/0xaa0 kernel/bpf/syscall.c:799 __do_sys_bpf kernel/bpf/syscall.c:2363 [inline] __se_sys_bpf kernel/bpf/syscall.c:2334 [inline] __x64_sys_bpf+0x208/0x380 kernel/bpf/syscall.c:2334 do_syscall_64+0xd0/0x4d0 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe Freed by task 8651: save_stack+0x43/0xd0 mm/kasan/kasan.c:448 set_track mm/kasan/kasan.c:460 [inline] __kasan_slab_free+0x102/0x150 mm/kasan/kasan.c:521 kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 __cache_free mm/slab.c:3498 [inline] kfree+0xcf/0x270 mm/slab.c:3813 smap_gc_work+0x76b/0xa80 kernel/bpf/sockmap.c:1594 process_one_work+0x830/0x1650 kernel/workqueue.c:2153 worker_thread+0x85/0xb60 kernel/workqueue.c:2296 kthread+0x324/0x3e0 kernel/kthread.c:246 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:413 The buggy address belongs to the object at ffff8800731de4c0 which belongs to the cache kmalloc-1024 of size 1024 The buggy address is located 584 bytes inside of 1024-byte region [ffff8800731de4c0, ffff8800731de8c0) The buggy address belongs to the page: page:ffffea0001cc7780 count:1 mapcount:0 mapping:ffff8800aa800ac0 index:0xffff8800731df6c0 compound_mapcount: 0 flags: 0x1fffc0000008100(slab|head) raw: 01fffc0000008100 ffffea00022a4b88 ffffea000286a088 ffff8800aa800ac0 raw: ffff8800731df6c0 ffff8800731de040 0000000100000004 0000000000000000 page dumped because: kasan: bad access detected Memory state around the buggy address: ffff8800731de600: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ffff8800731de680: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >ffff8800731de700: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ^ ffff8800731de780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ffff8800731de800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ==================================================================