bisecting cause commit starting from 68cc2999f6926590c7783f2de12ba467ecad8c7d building syzkaller on 2c86e0a54aaace4acb1b63de5d5eca42d49f51e0 testing commit 68cc2999f6926590c7783f2de12ba467ecad8c7d with gcc (GCC) 8.1.0 all runs: crashed: KASAN: use-after-free Write in skb_release_data testing release v5.0 testing commit 1c163f4c7b3f621efff9b28a47abb36f7378d783 with gcc (GCC) 8.1.0 all runs: OK # git bisect start 68cc2999f6926590c7783f2de12ba467ecad8c7d v5.0 Bisecting: 5954 revisions left to test after this (roughly 13 steps) [e0f0ae838a25464179d37f355d763f9ec139fc15] iio: adc: fix warning in Qualcomm PM8xxx HK/XOADC driver testing commit e0f0ae838a25464179d37f355d763f9ec139fc15 with gcc (GCC) 8.1.0 all runs: OK # git bisect good e0f0ae838a25464179d37f355d763f9ec139fc15 Bisecting: 3001 revisions left to test after this (roughly 12 steps) [4221b807d1f73c03d22543416d303b60a5d1ef31] Merge tag 'for-5.1/libata-20190301' of git://git.kernel.dk/linux-block testing commit 4221b807d1f73c03d22543416d303b60a5d1ef31 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 4221b807d1f73c03d22543416d303b60a5d1ef31 Bisecting: 1507 revisions left to test after this (roughly 11 steps) [45ba8d5d061b13494c2a7a7652d51b9da3d9e77a] Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost testing commit 45ba8d5d061b13494c2a7a7652d51b9da3d9e77a with gcc (GCC) 8.1.0 all runs: OK # git bisect good 45ba8d5d061b13494c2a7a7652d51b9da3d9e77a Bisecting: 743 revisions left to test after this (roughly 10 steps) [2b0a80b0d0bb0a3db74588279bf851b28c6c4705] Merge tag 'ceph-for-5.1-rc1' of git://github.com/ceph/ceph-client testing commit 2b0a80b0d0bb0a3db74588279bf851b28c6c4705 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 2b0a80b0d0bb0a3db74588279bf851b28c6c4705 Bisecting: 416 revisions left to test after this (roughly 9 steps) [2f194646fecaa9fd4607b670ee9ef84d9ed04566] Merge tag 'rproc-v5.1' of git://github.com/andersson/remoteproc testing commit 2f194646fecaa9fd4607b670ee9ef84d9ed04566 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 2f194646fecaa9fd4607b670ee9ef84d9ed04566 Bisecting: 203 revisions left to test after this (roughly 8 steps) [0aedadcf6b4863a0d6eaad05a26425cc52944027] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf testing commit 0aedadcf6b4863a0d6eaad05a26425cc52944027 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 0aedadcf6b4863a0d6eaad05a26425cc52944027 Bisecting: 101 revisions left to test after this (roughly 7 steps) [77d5ad4048fba5bd6e16f78498d4b41e5534b8f5] tipc: fix use-after-free in tipc_sk_filter_rcv testing commit 77d5ad4048fba5bd6e16f78498d4b41e5534b8f5 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 77d5ad4048fba5bd6e16f78498d4b41e5534b8f5 Bisecting: 50 revisions left to test after this (roughly 6 steps) [d84b899a946ea590355ab5f9ce13f56767ec312d] ice: Don't let VF know that it is untrusted testing commit d84b899a946ea590355ab5f9ce13f56767ec312d with gcc (GCC) 8.1.0 all runs: OK # git bisect good d84b899a946ea590355ab5f9ce13f56767ec312d Bisecting: 23 revisions left to test after this (roughly 5 steps) [d64fee0a0320ecc678903c30c2fed56b68979011] Merge tag 'mlx5-updates-2019-03-20' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux testing commit d64fee0a0320ecc678903c30c2fed56b68979011 with gcc (GCC) 8.1.0 all runs: crashed: KASAN: use-after-free Write in skb_release_data # git bisect bad d64fee0a0320ecc678903c30c2fed56b68979011 Bisecting: 13 revisions left to test after this (roughly 4 steps) [bdc837eecf73c391f5a9f97b5b61e6a1f30cf31f] net/mlx5e: Support VLAN modify action testing commit bdc837eecf73c391f5a9f97b5b61e6a1f30cf31f with gcc (GCC) 8.1.0 all runs: OK # git bisect good bdc837eecf73c391f5a9f97b5b61e6a1f30cf31f Bisecting: 6 revisions left to test after this (roughly 3 steps) [dc05360fee660a9dbe59824b3f7896534210432b] net: convert rps_needed and rfs_needed to new static branch api testing commit dc05360fee660a9dbe59824b3f7896534210432b with gcc (GCC) 8.1.0 all runs: OK # git bisect good dc05360fee660a9dbe59824b3f7896534210432b Bisecting: 3 revisions left to test after this (roughly 2 steps) [bdaba8959e9248524f3d148d1aa47f13944ba8e8] Merge branch 'tcp-rx-tx-cache' testing commit bdaba8959e9248524f3d148d1aa47f13944ba8e8 with gcc (GCC) 8.1.0 all runs: crashed: KASAN: use-after-free Write in skb_release_data # git bisect bad bdaba8959e9248524f3d148d1aa47f13944ba8e8 Bisecting: 0 revisions left to test after this (roughly 1 step) [8b27dae5a2e89a61c46c6dbc76c040c0e6d0ed4c] tcp: add one skb cache for rx testing commit 8b27dae5a2e89a61c46c6dbc76c040c0e6d0ed4c with gcc (GCC) 8.1.0 all runs: crashed: KASAN: use-after-free Write in skb_release_data # git bisect bad 8b27dae5a2e89a61c46c6dbc76c040c0e6d0ed4c Bisecting: 0 revisions left to test after this (roughly 0 steps) [472c2e07eef045145bc1493cc94a01c87140780a] tcp: add one skb cache for tx testing commit 472c2e07eef045145bc1493cc94a01c87140780a with gcc (GCC) 8.1.0 all runs: crashed: KASAN: use-after-free Write in skb_release_data # git bisect bad 472c2e07eef045145bc1493cc94a01c87140780a 472c2e07eef045145bc1493cc94a01c87140780a is the first bad commit commit 472c2e07eef045145bc1493cc94a01c87140780a Author: Eric Dumazet Date: Fri Mar 22 08:56:39 2019 -0700 tcp: add one skb cache for tx On hosts with a lot of cores, RPC workloads suffer from heavy contention on slab spinlocks. 20.69% [kernel] [k] queued_spin_lock_slowpath 5.64% [kernel] [k] _raw_spin_lock 3.83% [kernel] [k] syscall_return_via_sysret 3.48% [kernel] [k] __entry_text_start 1.76% [kernel] [k] __netif_receive_skb_core 1.64% [kernel] [k] __fget For each sendmsg(), we allocate one skb, and free it at the time ACK packet comes. In many cases, ACK packets are handled by another cpus, and this unfortunately incurs heavy costs for slab layer. This patch uses an extra pointer in socket structure, so that we try to reuse the same skb and avoid these expensive costs. We cache at most one skb per socket so this should be safe as far as memory pressure is concerned. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Acked-by: Willem de Bruijn Signed-off-by: David S. Miller include/net/sock.h | 5 +++++ net/ipv4/tcp.c | 50 +++++++++++++++++++++++--------------------------- 2 files changed, 28 insertions(+), 27 deletions(-) revisions tested: 16, total time: 3h26m37.226791457s (build: 1h24m37.487380919s, test: 1h58m35.218675051s) first bad commit: 472c2e07eef045145bc1493cc94a01c87140780a tcp: add one skb cache for tx cc: ["davem@davemloft.net" "edumazet@google.com" "soheil@google.com" "willemb@google.com"] crash: KASAN: use-after-free Write in skb_release_data ================================================================== BUG: KASAN: use-after-free in atomic_sub_return include/asm-generic/atomic-instrumented.h:159 [inline] BUG: KASAN: use-after-free in skb_release_data+0xe4/0x680 net/core/skbuff.c:566 Write of size 4 at addr ffff88807781e320 by task syz-executor.3/7565 CPU: 1 PID: 7565 Comm: syz-executor.3 Not tainted 5.0.0+ #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x113/0x167 lib/dump_stack.c:113 print_address_description.cold.5+0x9/0x1ff mm/kasan/report.c:187 kasan_report.cold.6+0x1b/0x39 mm/kasan/report.c:317 check_memory_region_inline mm/kasan/generic.c:185 [inline] check_memory_region+0x13e/0x1b0 mm/kasan/generic.c:191 kasan_check_write+0x14/0x20 mm/kasan/common.c:108 atomic_sub_return include/asm-generic/atomic-instrumented.h:159 [inline] skb_release_data+0xe4/0x680 net/core/skbuff.c:566 skb_release_all+0x3d/0x50 net/core/skbuff.c:631 __kfree_skb net/core/skbuff.c:645 [inline] kfree_skb+0x97/0x270 net/core/skbuff.c:663 skb_queue_purge+0x12/0x30 net/core/skbuff.c:2906 packet_release+0x7c2/0xae0 net/packet/af_packet.c:3026 __sock_release+0xc7/0x2a0 net/socket.c:599 sock_close+0x10/0x20 net/socket.c:1247 __fput+0x252/0x800 fs/file_table.c:278 ____fput+0x9/0x10 fs/file_table.c:309 task_work_run+0x10e/0x190 kernel/task_work.c:113 exit_task_work include/linux/task_work.h:22 [inline] do_exit+0x9ca/0x2e80 kernel/exit.c:876 do_group_exit+0xf4/0x2f0 kernel/exit.c:980 get_signal+0x2e7/0x1840 kernel/signal.c:2575 do_signal+0x87/0x1930 arch/x86/kernel/signal.c:816 exit_to_usermode_loop+0x114/0x200 arch/x86/entry/common.c:162 prepare_exit_to_usermode arch/x86/entry/common.c:197 [inline] syscall_return_slowpath arch/x86/entry/common.c:268 [inline] do_syscall_64+0x40d/0x4e0 arch/x86/entry/common.c:293 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x457879 Code: 8d b5 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 5b b5 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f00c36d6cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: 0000000000000000 RBX: 000000000071bf08 RCX: 0000000000457879 RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000000071bf08 RBP: 000000000071bf00 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000000000071bf0c R13: 00007ffe0be894bf R14: 00007f00c36d7700 R15: 0000000000000000 Allocated by task 7565: save_stack+0x43/0xd0 mm/kasan/common.c:75 set_track mm/kasan/common.c:87 [inline] __kasan_kmalloc.constprop.9+0xc7/0xd0 mm/kasan/common.c:497 kasan_kmalloc+0x9/0x10 mm/kasan/common.c:511 __do_kmalloc_node mm/slab.c:3686 [inline] __kmalloc_node_track_caller+0x4d/0x70 mm/slab.c:3700 __kmalloc_reserve.isra.38+0x2c/0xc0 net/core/skbuff.c:140 __alloc_skb+0xd7/0x570 net/core/skbuff.c:208 alloc_skb_fclone include/linux/skbuff.h:1107 [inline] sk_stream_alloc_skb+0x131/0xc80 net/ipv4/tcp.c:889 tcp_connect+0xd78/0x3fd0 net/ipv4/tcp_output.c:3521 tcp_v4_connect+0x1ac9/0x21b0 net/ipv4/tcp_ipv4.c:315 __inet_stream_connect+0x66b/0xd20 net/ipv4/af_inet.c:655 tcp_sendmsg_fastopen net/ipv4/tcp.c:1158 [inline] tcp_sendmsg_locked+0x20cb/0x36a0 net/ipv4/tcp.c:1200 tcp_sendmsg+0x27/0x40 net/ipv4/tcp.c:1434 inet_sendmsg+0x10d/0x460 net/ipv4/af_inet.c:798 sock_sendmsg_nosec net/socket.c:651 [inline] sock_sendmsg+0xb7/0xf0 net/socket.c:661 __sys_sendto+0x1f2/0x2e0 net/socket.c:1932 __do_sys_sendto net/socket.c:1944 [inline] __se_sys_sendto net/socket.c:1940 [inline] __x64_sys_sendto+0xdc/0x1a0 net/socket.c:1940 do_syscall_64+0xd6/0x4e0 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe Freed by task 7565: save_stack+0x43/0xd0 mm/kasan/common.c:75 set_track mm/kasan/common.c:87 [inline] __kasan_slab_free+0x102/0x150 mm/kasan/common.c:459 kasan_slab_free+0xe/0x10 mm/kasan/common.c:467 __cache_free mm/slab.c:3498 [inline] kfree+0xcf/0x230 mm/slab.c:3821 skb_free_head+0x6e/0x90 net/core/skbuff.c:557 skb_release_data+0x478/0x680 net/core/skbuff.c:577 skb_release_all+0x3d/0x50 net/core/skbuff.c:631 __kfree_skb net/core/skbuff.c:645 [inline] kfree_skb+0x97/0x270 net/core/skbuff.c:663 skb_queue_purge+0x12/0x30 net/core/skbuff.c:2906 packet_release+0x7c2/0xae0 net/packet/af_packet.c:3026 __sock_release+0xc7/0x2a0 net/socket.c:599 sock_close+0x10/0x20 net/socket.c:1247 __fput+0x252/0x800 fs/file_table.c:278 ____fput+0x9/0x10 fs/file_table.c:309 task_work_run+0x10e/0x190 kernel/task_work.c:113 exit_task_work include/linux/task_work.h:22 [inline] do_exit+0x9ca/0x2e80 kernel/exit.c:876 do_group_exit+0xf4/0x2f0 kernel/exit.c:980 get_signal+0x2e7/0x1840 kernel/signal.c:2575 do_signal+0x87/0x1930 arch/x86/kernel/signal.c:816 exit_to_usermode_loop+0x114/0x200 arch/x86/entry/common.c:162 prepare_exit_to_usermode arch/x86/entry/common.c:197 [inline] syscall_return_slowpath arch/x86/entry/common.c:268 [inline] do_syscall_64+0x40d/0x4e0 arch/x86/entry/common.c:293 entry_SYSCALL_64_after_hwframe+0x49/0xbe The buggy address belongs to the object at ffff88807781e040 which belongs to the cache kmalloc-1k of size 1024 The buggy address is located 736 bytes inside of 1024-byte region [ffff88807781e040, ffff88807781e440) The buggy address belongs to the page: page:ffffea0001de0780 count:1 mapcount:0 mapping:ffff88802d400ac0 index:0x0 compound_mapcount: 0 flags: 0x5fffc0000010200(slab|head) raw: 05fffc0000010200 ffffea0001f7fa88 ffffea0001d2db88 ffff88802d400ac0 raw: 0000000000000000 ffff88807781e040 0000000100000007 0000000000000000 page dumped because: kasan: bad access detected Memory state around the buggy address: ffff88807781e200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ffff88807781e280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >ffff88807781e300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ^ ffff88807781e380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ffff88807781e400: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc ==================================================================