bisecting cause commit starting from 13fac1d851e09109096b5862bf37c3da6908fb48 building syzkaller on e7caca8e1ed4acd5ba185b96aea8a400cf1717b4 testing commit 13fac1d851e09109096b5862bf37c3da6908fb48 with gcc (GCC) 8.1.0 kernel signature: e870a994c00013e51838e92c3729002142dece1f5893fdb01d00cd6e2ece2245 all runs: crashed: WARNING in bpf_check testing release v5.5 testing commit d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 with gcc (GCC) 8.1.0 kernel signature: a6b42b21161b43e6269b41e9efd6894ee359b7b8d8e34fd4868550ce87a0cead all runs: OK # git bisect start 13fac1d851e09109096b5862bf37c3da6908fb48 d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 Bisecting: 5585 revisions left to test after this (roughly 13 steps) [9f68e3655aae6d49d6ba05dd263f99f33c2567af] Merge tag 'drm-next-2020-01-30' of git://anongit.freedesktop.org/drm/drm testing commit 9f68e3655aae6d49d6ba05dd263f99f33c2567af with gcc (GCC) 8.1.0 kernel signature: 30041f79edbf8de41ea4e753e049e08698af2f26cd7be4343b83a5d8f8b345e0 all runs: OK # git bisect good 9f68e3655aae6d49d6ba05dd263f99f33c2567af Bisecting: 2831 revisions left to test after this (roughly 12 steps) [469030d454bd1620c7b2651d9ec8cdcbaa74deb9] Merge tag 'armsoc-soc' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc testing commit 469030d454bd1620c7b2651d9ec8cdcbaa74deb9 with gcc (GCC) 8.1.0 kernel signature: f88368fbf45a2fc47b1b63eaff2026a9ab1071831f588d371c92edf107ee88a3 all runs: OK # git bisect good 469030d454bd1620c7b2651d9ec8cdcbaa74deb9 Bisecting: 1415 revisions left to test after this (roughly 11 steps) [7a47281439ba00b11fc098f36695522184ce5a82] net: sched: lock action when translating it to flow_action infra testing commit 7a47281439ba00b11fc098f36695522184ce5a82 with gcc (GCC) 8.1.0 kernel signature: a75ab6ed7a75f095cfda5dc098a7f9897e44970f7564510edf5ca9fbd1a79b25 all runs: OK # git bisect good 7a47281439ba00b11fc098f36695522184ce5a82 Bisecting: 707 revisions left to test after this (roughly 10 steps) [b7331aa204a1b3417d0485bad9380ad7360558d5] net/mlx5: Add fsm_reactivate callback support testing commit b7331aa204a1b3417d0485bad9380ad7360558d5 with gcc (GCC) 8.1.0 kernel signature: 41d289894bffc05cce599f7bc61f50dd9a4b88dff47c4378826d50d3cf16c83a all runs: OK # git bisect good b7331aa204a1b3417d0485bad9380ad7360558d5 Bisecting: 431 revisions left to test after this (roughly 9 steps) [7058b837899fc978c9f8a033fa29ab07360a85c8] Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net testing commit 7058b837899fc978c9f8a033fa29ab07360a85c8 with gcc (GCC) 8.1.0 kernel signature: 6d0a605345a14a025a3b9070811a1b74933f843cf39fb8fbc0342d30ae71c309 all runs: OK # git bisect good 7058b837899fc978c9f8a033fa29ab07360a85c8 Bisecting: 215 revisions left to test after this (roughly 8 steps) [05ef614c559ee8a496d590f0f468f9f883a06ca9] selftests: mlxsw: Use busywait helper in vxlan test testing commit 05ef614c559ee8a496d590f0f468f9f883a06ca9 with gcc (GCC) 8.1.0 kernel signature: 58ca7b81ec5dd3f0e5112e37b078e8a8eb61e61cbf273ec56a197caa33762959 all runs: OK # git bisect good 05ef614c559ee8a496d590f0f468f9f883a06ca9 Bisecting: 105 revisions left to test after this (roughly 7 steps) [9f0ca0c1a50a7de5c71970aa452941199ed210d9] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next testing commit 9f0ca0c1a50a7de5c71970aa452941199ed210d9 with gcc (GCC) 8.1.0 kernel signature: 0e64ed60d29a609d2b36ad5c0a7173e1751de3e25a86cb22e7a6c38512a74736 all runs: crashed: WARNING in bpf_check # git bisect bad 9f0ca0c1a50a7de5c71970aa452941199ed210d9 Bisecting: 54 revisions left to test after this (roughly 6 steps) [af71b090c88c12816d43514190790de919921cea] l2tp: Replace zero-length array with flexible-array member testing commit af71b090c88c12816d43514190790de919921cea with gcc (GCC) 8.1.0 kernel signature: d6d2e88f5b6b4eb5b4f295196132cb6f12cb9d3bfec333e48df14b42706c2e4e all runs: OK # git bisect good af71b090c88c12816d43514190790de919921cea Bisecting: 27 revisions left to test after this (roughly 5 steps) [80a836c2506b2b249a9934fbe373eb7a4a98db86] Merge branch 'BPF_and_RT' testing commit 80a836c2506b2b249a9934fbe373eb7a4a98db86 with gcc (GCC) 8.1.0 kernel signature: 123aa5f6d964224a65d41ae64ca9f3eac1b01a3d34904cfa47c21eb43dd73116 all runs: crashed: WARNING in bpf_check # git bisect bad 80a836c2506b2b249a9934fbe373eb7a4a98db86 Bisecting: 13 revisions left to test after this (roughly 4 steps) [569de905ebc30b9c61be7aa557403aeb5a9141a4] bpf: Dont iterate over possible CPUs with interrupts disabled testing commit 569de905ebc30b9c61be7aa557403aeb5a9141a4 with gcc (GCC) 8.1.0 kernel signature: c44ebcba64a6ca430d47abf742d94f9e9172b5b45967c5bc695ed012405373c3 all runs: crashed: WARNING in bpf_check # git bisect bad 569de905ebc30b9c61be7aa557403aeb5a9141a4 Bisecting: 6 revisions left to test after this (roughly 3 steps) [dbca151cad736c99f4817076daf9fd02ed0c2daa] bpf: Update locking comment in hashtab code testing commit dbca151cad736c99f4817076daf9fd02ed0c2daa with gcc (GCC) 8.1.0 kernel signature: a4ba7ce2d28ef6fff8679a810c54f10471d4ed19a1f8aad4aa7550ffe3ed76cc all runs: crashed: WARNING in bpf_check # git bisect bad dbca151cad736c99f4817076daf9fd02ed0c2daa Bisecting: 2 revisions left to test after this (roughly 2 steps) [8eece07c011f88da0ccf4127fca9a4e4faaf58ae] Merge tag 'sched-for-bpf-2020-02-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into bpf-next testing commit 8eece07c011f88da0ccf4127fca9a4e4faaf58ae with gcc (GCC) 8.1.0 kernel signature: 4cbd5ccb1965023b9b9d861e4e8e992c4e0685444655765a544de0110581c521 all runs: OK # git bisect good 8eece07c011f88da0ccf4127fca9a4e4faaf58ae Bisecting: 0 revisions left to test after this (roughly 1 step) [2ed905c521e56aead6987df94c083efb0ee59895] bpf: Enforce preallocation for instrumentation programs on RT testing commit 2ed905c521e56aead6987df94c083efb0ee59895 with gcc (GCC) 8.1.0 kernel signature: ffa930854ecb585882ac807b9a30fe32aadf184d60f4be9e0e5071fa15c1625b all runs: crashed: WARNING in bpf_check # git bisect bad 2ed905c521e56aead6987df94c083efb0ee59895 Bisecting: 0 revisions left to test after this (roughly 0 steps) [94dacdbd5d2dfa2cffd308f128d78c99f855f5be] bpf: Tighten the requirements for preallocated hash maps testing commit 94dacdbd5d2dfa2cffd308f128d78c99f855f5be with gcc (GCC) 8.1.0 kernel signature: 603d9f82d791a136e128547908085bc5e720c98fd61007fc3975dbc911b34899 all runs: crashed: WARNING in bpf_check # git bisect bad 94dacdbd5d2dfa2cffd308f128d78c99f855f5be 94dacdbd5d2dfa2cffd308f128d78c99f855f5be is the first bad commit commit 94dacdbd5d2dfa2cffd308f128d78c99f855f5be Author: Thomas Gleixner Date: Mon Feb 24 15:01:32 2020 +0100 bpf: Tighten the requirements for preallocated hash maps The assumption that only programs attached to perf NMI events can deadlock on memory allocators is wrong. Assume the following simplified callchain: kmalloc() from regular non BPF context cache empty freelist empty lock(zone->lock); tracepoint or kprobe BPF() update_elem() lock(bucket) kmalloc() cache empty freelist empty lock(zone->lock); <- DEADLOCK There are other ways which do not involve locking to create wreckage: kmalloc() from regular non BPF context local_irq_save(); ... obj = slab_first(); kprobe() BPF() update_elem() lock(bucket) kmalloc() local_irq_save(); ... obj = slab_first(); <- Same object as above ... So preallocation _must_ be enforced for all variants of intrusive instrumentation. Unfortunately immediate enforcement would break backwards compatibility, so for now such programs still are allowed to run, but a one time warning is emitted in dmesg and the verifier emits a warning in the verifier log as well so developers are made aware about this and can fix their programs before the enforcement becomes mandatory. Signed-off-by: Thomas Gleixner Signed-off-by: Alexei Starovoitov Link: https://lore.kernel.org/bpf/20200224145642.540542802@linutronix.de kernel/bpf/verifier.c | 39 ++++++++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 11 deletions(-) culprit signature: 603d9f82d791a136e128547908085bc5e720c98fd61007fc3975dbc911b34899 parent signature: 4cbd5ccb1965023b9b9d861e4e8e992c4e0685444655765a544de0110581c521 revisions tested: 16, total time: 3h55m52.233369554s (build: 1h51m11.429645646s, test: 2h3m30.024752099s) first bad commit: 94dacdbd5d2dfa2cffd308f128d78c99f855f5be bpf: Tighten the requirements for preallocated hash maps cc: ["andriin@fb.com" "ast@kernel.org" "bpf@vger.kernel.org" "daniel@iogearbox.net" "kafai@fb.com" "linux-kernel@vger.kernel.org" "netdev@vger.kernel.org" "songliubraving@fb.com" "tglx@linutronix.de" "yhs@fb.com"] crash: WARNING in bpf_check ------------[ cut here ]------------ trace type BPF program uses run-time allocation WARNING: CPU: 0 PID: 8508 at kernel/bpf/verifier.c:8181 check_map_prog_compatibility kernel/bpf/verifier.c:8181 [inline] WARNING: CPU: 0 PID: 8508 at kernel/bpf/verifier.c:8181 replace_map_fd_with_map_ptr kernel/bpf/verifier.c:8276 [inline] WARNING: CPU: 0 PID: 8508 at kernel/bpf/verifier.c:8181 bpf_check+0x5ca1/0x91af kernel/bpf/verifier.c:10082 Kernel panic - not syncing: panic_on_warn set ... CPU: 0 PID: 8508 Comm: syz-executor.3 Not tainted 5.6.0-rc2-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x128/0x182 lib/dump_stack.c:118 panic+0x22a/0x4e3 kernel/panic.c:221 __warn.cold.10+0x25/0x26 kernel/panic.c:582 report_bug+0x1ad/0x270 lib/bug.c:195 fixup_bug arch/x86/kernel/traps.c:174 [inline] do_error_trap+0x123/0x210 arch/x86/kernel/traps.c:267 do_invalid_op+0x31/0x40 arch/x86/kernel/traps.c:286 invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1027 RIP: 0010:check_map_prog_compatibility kernel/bpf/verifier.c:8181 [inline] RIP: 0010:replace_map_fd_with_map_ptr kernel/bpf/verifier.c:8276 [inline] RIP: 0010:bpf_check+0x5ca1/0x91af kernel/bpf/verifier.c:10082 Code: 70 87 48 89 95 e8 fd ff ff 44 89 8d f0 fd ff ff 4c 89 85 f8 fd ff ff 48 89 85 18 fe ff ff c6 05 ce dc 1f 08 01 e8 cb c2 c7 ff <0f> 0b 48 8b 95 e8 fd ff ff 44 8b 8d f0 fd ff ff 4c 8b 85 f8 fd ff RSP: 0018:ffffc90002df77f8 EFLAGS: 00010286 RAX: 0000000000000000 RBX: ffffc900011a6048 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000007 RDI: ffffffff8b6563a0 RBP: ffffc90002df7a38 R08: ffffed1015d045c9 R09: ffffed1015d045c9 R10: ffffed1015d045c8 R11: ffff8880ae822e43 R12: ffff8880a450b800 R13: ffff888093afc000 R14: ffffc900011a6049 R15: dffffc0000000000 bpf_prog_load+0xd9c/0x1280 kernel/bpf/syscall.c:2101 __do_sys_bpf+0xfc2/0x32f0 kernel/bpf/syscall.c:3396 do_syscall_64+0xc6/0x5e0 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x45c679 Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007efe93befc78 EFLAGS: 00000246 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 00007efe93bf06d4 RCX: 000000000045c679 RDX: 0000000000000014 RSI: 0000000020fed000 RDI: 0000000000000005 RBP: 000000000076bf00 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff R13: 0000000000000068 R14: 00000000004c2eb9 R15: 000000000076bf0c Kernel Offset: disabled Rebooting in 86400 seconds..