ci2 starts bisection 2023-12-08 01:25:49.807297934 +0000 UTC m=+6706.913219284 bisecting fixing commit since d3212c2dbababf849d940f5f7001f4fde222b888 building syzkaller on 0b6a67ac4b0dc26f43030c5edd01c9175f13b784 ensuring issue is reproducible on original commit d3212c2dbababf849d940f5f7001f4fde222b888 testing commit d3212c2dbababf849d940f5f7001f4fde222b888 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: b4744d90bff032f7566c1f229adee50a067a636619fd9ec85cc7e70c7f75ffc3 all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] check whether we can drop unnecessary instrumentation disabling configs for [HANG LEAK UBSAN BUG LOCKDEP ATOMIC_SLEEP], they are not needed testing commit d3212c2dbababf849d940f5f7001f4fde222b888 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 698918d3deebdbbc16ccdf0bd5d56699e998221f7c48a528a3b1133ca28e4c5f all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] the bug reproduces without the instrumentation disabling configs for [LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN BUG], they are not needed kconfig minimization: base=5179 full=6487 leaves diff=250 split chunks (needed=false): <250> split chunk #0 of len 250 into 5 parts testing without sub-chunk 1/5 disabling configs for [ATOMIC_SLEEP HANG LEAK UBSAN BUG LOCKDEP], they are not needed testing commit d3212c2dbababf849d940f5f7001f4fde222b888 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ce214cb714f9ed8a626d477c45527e94215cbe0ac3037c7f9622051cf4b14ae5 all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [LEAK UBSAN BUG LOCKDEP ATOMIC_SLEEP HANG], they are not needed testing commit d3212c2dbababf849d940f5f7001f4fde222b888 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: f19bc4c06fffe48056b09f56f41f0ff316e905c742bd1bd80f8e6f7b7d471a7d all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] the chunk can be dropped testing without sub-chunk 3/5 disabling configs for [UBSAN BUG LOCKDEP ATOMIC_SLEEP HANG LEAK], they are not needed testing commit d3212c2dbababf849d940f5f7001f4fde222b888 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6c89d4daffc3deb631817521fc90ae697a78f7c4b47af25e9db73c74471e1702 all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] the chunk can be dropped testing without sub-chunk 4/5 disabling configs for [BUG LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing commit d3212c2dbababf849d940f5f7001f4fde222b888 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: b67aa12544825a870ad800eb83cfd62fc6bef5a065d918522fa5e70d91926efd all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [LEAK UBSAN BUG LOCKDEP ATOMIC_SLEEP HANG], they are not needed testing commit d3212c2dbababf849d940f5f7001f4fde222b888 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 failed building d3212c2dbababf849d940f5f7001f4fde222b888: net/socket.c:1225: undefined reference to `wext_handle_ioctl' net/socket.c:3420: undefined reference to `compat_wext_handle_ioctl' net/core/net-procfs.c:329: undefined reference to `wext_proc_init' net/core/net-procfs.c:345: undefined reference to `wext_proc_exit' minimized to 50 configs; suspects: [HID_ZEROPLUS USB_NET_CDC_MBIM USB_NET_CDC_SUBSET USB_NET_CDC_SUBSET_ENABLE USB_NET_DM9601 USB_NET_GL620A USB_NET_MCS7830 USB_NET_NET1080 USB_NET_PLUSB USB_NET_RNDIS_HOST USB_NET_SMSC75XX USB_NET_SMSC95XX USB_NET_SR9700 USB_NET_SR9800 USB_NET_ZAURUS USB_OHCI_HCD USB_OHCI_HCD_PCI USB_OHCI_HCD_PLATFORM USB_OTG USB_OTG_FSM USB_PRINTER USB_SERIAL_GENERIC USB_SERIAL_PL2303 USB_STORAGE_ALAUDA USB_STORAGE_CYPRESS_ATACB USB_STORAGE_DATAFAB USB_STORAGE_FREECOM USB_STORAGE_ISD200 USB_STORAGE_JUMPSHOT USB_STORAGE_KARMA USB_STORAGE_ONETOUCH USB_STORAGE_SDDR09 USB_STORAGE_SDDR55 USB_STORAGE_USBAT USB_TRANCEVIBRATOR USB_U_AUDIO USB_U_ETHER USB_U_SERIAL USB_WDM WLAN WLAN_VENDOR_ATH WLAN_VENDOR_ATMEL WLAN_VENDOR_BROADCOM WLAN_VENDOR_INTERSIL WLAN_VENDOR_MARVELL WLAN_VENDOR_MEDIATEK WLAN_VENDOR_MICROCHIP WLAN_VENDOR_PURELIFI WLAN_VENDOR_RALINK WLAN_VENDOR_REALTEK WLAN_VENDOR_RSI WLAN_VENDOR_SILABS WLAN_VENDOR_ZYDAS X86_X32_ABI ZEROPLUS_FF] disabling configs for [BUG LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing current HEAD c9b5c232e7152ac1ad5e61148b17588215bf5bdf testing commit c9b5c232e7152ac1ad5e61148b17588215bf5bdf gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 1be5b0795bb17e88ed437eb76832f456a16baa73183ef2f85e14a23cb42b8ccd all runs: OK false negative chance: 0.000 # git bisect start c9b5c232e7152ac1ad5e61148b17588215bf5bdf d3212c2dbababf849d940f5f7001f4fde222b888 Bisecting: 2874 revisions left to test after this (roughly 12 steps) [671486793f729fa8d79f5bd1f6224a2af00c2d63] selftests: mptcp: connect: fail if nft supposed to work determine whether the revision contains the guilty commit checking the merge base b1644a0031cfb3ca2cbd84c92f771f8ebb62302d no existing result, test the revision testing commit b1644a0031cfb3ca2cbd84c92f771f8ebb62302d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 723e2973020ea08dba318a1dc8ce4d72d385bc05016401a2673d8f0b00f716af all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] testing commit 671486793f729fa8d79f5bd1f6224a2af00c2d63 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 565ea5cb78b5bcc1a753cad6b4153fa0a575a1059fd40c95ac56124c284a1959 all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] # git bisect good 671486793f729fa8d79f5bd1f6224a2af00c2d63 Bisecting: 1437 revisions left to test after this (roughly 11 steps) [711fb92606208a8626b785da4f9f23d648a5b6c8] md/raid5-cache: fix null-ptr-deref for r5l_flush_stripe_to_raid() determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit 711fb92606208a8626b785da4f9f23d648a5b6c8 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 0d594a1a07c9fe5586f1cb0a5bada16be6429b48973079ac2ee100abff42d563 all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] # git bisect good 711fb92606208a8626b785da4f9f23d648a5b6c8 Bisecting: 718 revisions left to test after this (roughly 10 steps) [2f0acb0736ecc3eb85dc80ad2790d634dcb10b58] team: fix null-ptr-deref when team device type is changed determine whether the revision contains the guilty commit revision 711fb92606208a8626b785da4f9f23d648a5b6c8 crashed and is reachable testing commit 2f0acb0736ecc3eb85dc80ad2790d634dcb10b58 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 61b3070f5834a5f0fdde4b149dc7a6a8f5fc4fcb19075943b038ce8ea226ff24 all runs: OK false negative chance: 0.000 # git bisect bad 2f0acb0736ecc3eb85dc80ad2790d634dcb10b58 Bisecting: 359 revisions left to test after this (roughly 9 steps) [b7cbcafb6d04b95291d1b9e615879b5ae7c37ebf] perf vendor events: Update the JSON/events descriptions for power10 platform determine whether the revision contains the guilty commit revision 671486793f729fa8d79f5bd1f6224a2af00c2d63 crashed and is reachable testing commit b7cbcafb6d04b95291d1b9e615879b5ae7c37ebf gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: fd6ccf8c6417706a47db698699a16539d70995122141b6e5cf386d276e39584d all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] # git bisect good b7cbcafb6d04b95291d1b9e615879b5ae7c37ebf Bisecting: 179 revisions left to test after this (roughly 8 steps) [d7b0fe3487d203c04ee1bda91a63bd4dd398c350] wifi: cfg80211: ocb: don't leave if not joined determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit d7b0fe3487d203c04ee1bda91a63bd4dd398c350 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 24ca0eb33f2bf2e6728102d3e1ae5eb5ff7d874c4fdaea85f3c82a278c27bc62 all runs: OK false negative chance: 0.000 # git bisect bad d7b0fe3487d203c04ee1bda91a63bd4dd398c350 Bisecting: 89 revisions left to test after this (roughly 7 steps) [7c6ba20a0b9aeb82a6c097c74ccbecdda8e9fc25] mtd: spi-nor: Correct flags for Winbond w25q128 determine whether the revision contains the guilty commit revision 671486793f729fa8d79f5bd1f6224a2af00c2d63 crashed and is reachable testing commit 7c6ba20a0b9aeb82a6c097c74ccbecdda8e9fc25 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 17f0a9225c6c107abf5152bbe23fb66d42c121a4423473bb3258b510a43d31d9 all runs: OK false negative chance: 0.000 # git bisect bad 7c6ba20a0b9aeb82a6c097c74ccbecdda8e9fc25 Bisecting: 44 revisions left to test after this (roughly 6 steps) [54b59bc18d195e560934ff183d4cc48b2deb019b] ip_tunnels: use DEV_STATS_INC() determine whether the revision contains the guilty commit revision 671486793f729fa8d79f5bd1f6224a2af00c2d63 crashed and is reachable testing commit 54b59bc18d195e560934ff183d4cc48b2deb019b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6fcb25b3c164e3d7f458480a932fa08cb175a8e6f10270d999f338a96c3e4857 all runs: OK false negative chance: 0.000 # git bisect bad 54b59bc18d195e560934ff183d4cc48b2deb019b Bisecting: 22 revisions left to test after this (roughly 5 steps) [f1175881ddd91db46648c28eec05628e30e14c65] net: use sk_forward_alloc_get() in sk_get_meminfo() determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit f1175881ddd91db46648c28eec05628e30e14c65 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d818869f88fe833492da269cb2e8e438a6aa3a9ba639242173e701dd89605bb5 run #0: crashed: KASAN: use-after-free Read in consume_skb run #1: crashed: KASAN: use-after-free Read in skb_dequeue run #2: crashed: KASAN: use-after-free Read in consume_skb run #3: crashed: KASAN: use-after-free Read in consume_skb run #4: crashed: KASAN: use-after-free Read in consume_skb run #5: crashed: KASAN: use-after-free Read in consume_skb run #6: crashed: KASAN: use-after-free Read in consume_skb run #7: crashed: KASAN: use-after-free Read in consume_skb run #8: crashed: KASAN: use-after-free Read in consume_skb run #9: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] # git bisect good f1175881ddd91db46648c28eec05628e30e14c65 Bisecting: 11 revisions left to test after this (roughly 4 steps) [b9cdbb38e030fc2fe97fe27b54cbb6b4fbff250f] af_unix: Fix data-races around user->unix_inflight. determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit b9cdbb38e030fc2fe97fe27b54cbb6b4fbff250f gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8252153f1a1226ae64bd8465727cd7f70b66e0ba84e6458013f9f76e62b08734 all runs: OK false negative chance: 0.000 # git bisect bad b9cdbb38e030fc2fe97fe27b54cbb6b4fbff250f Bisecting: 5 revisions left to test after this (roughly 3 steps) [24b1e835db343cc410c1270cdb95f723b282825d] igb: disable virtualization features on 82580 determine whether the revision contains the guilty commit revision f1175881ddd91db46648c28eec05628e30e14c65 crashed and is reachable testing commit 24b1e835db343cc410c1270cdb95f723b282825d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d7afd18fda53f0cf70c9d7a3b6c6293ac21d13ae9564963919e23276c8fdff6d all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] # git bisect good 24b1e835db343cc410c1270cdb95f723b282825d Bisecting: 2 revisions left to test after this (roughly 2 steps) [aa8fd3a636918390b8040cb1d646dbf8214b8ed7] net: ipv6/addrconf: avoid integer underflow in ipv6_create_tempaddr determine whether the revision contains the guilty commit revision 671486793f729fa8d79f5bd1f6224a2af00c2d63 crashed and is reachable testing commit aa8fd3a636918390b8040cb1d646dbf8214b8ed7 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 31aaf28272814a419e870cf0dc6cdf46797114690c1662fff3ebc30ee1781112 all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] # git bisect good aa8fd3a636918390b8040cb1d646dbf8214b8ed7 Bisecting: 0 revisions left to test after this (roughly 1 step) [923877254f002ae87d441382bb1096d9e773d56d] bpf, sockmap: Fix skb refcnt race after locking changes determine whether the revision contains the guilty commit revision 671486793f729fa8d79f5bd1f6224a2af00c2d63 crashed and is reachable testing commit 923877254f002ae87d441382bb1096d9e773d56d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: eb507a9005b343c1475941eb7e424e73ca6caff84b424440cd41fe0af47feac9 all runs: OK false negative chance: 0.000 # git bisect bad 923877254f002ae87d441382bb1096d9e773d56d Bisecting: 0 revisions left to test after this (roughly 0 steps) [71fb38b222cff65f282b068f9ef83d5ce5e9c87d] net: phy: micrel: Correct bit assignments for phy_device flags determine whether the revision contains the guilty commit revision b7cbcafb6d04b95291d1b9e615879b5ae7c37ebf crashed and is reachable testing commit 71fb38b222cff65f282b068f9ef83d5ce5e9c87d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 86d256ac30b4b29de1b3f6f0dabf23efe95d4887322b76398261e4f72b2ce046 all runs: crashed: KASAN: use-after-free Read in consume_skb representative crash: KASAN: use-after-free Read in consume_skb, types: [KASAN] # git bisect good 71fb38b222cff65f282b068f9ef83d5ce5e9c87d 923877254f002ae87d441382bb1096d9e773d56d is the first bad commit commit 923877254f002ae87d441382bb1096d9e773d56d Author: John Fastabend Date: Fri Sep 1 13:21:37 2023 -0700 bpf, sockmap: Fix skb refcnt race after locking changes [ Upstream commit a454d84ee20baf7bd7be90721b9821f73c7d23d9 ] There is a race where skb's from the sk_psock_backlog can be referenced after userspace side has already skb_consumed() the sk_buff and its refcnt dropped to zer0 causing use after free. The flow is the following: while ((skb = skb_peek(&psock->ingress_skb)) sk_psock_handle_Skb(psock, skb, ..., ingress) if (!ingress) ... sk_psock_skb_ingress sk_psock_skb_ingress_enqueue(skb) msg->skb = skb sk_psock_queue_msg(psock, msg) skb_dequeue(&psock->ingress_skb) The sk_psock_queue_msg() puts the msg on the ingress_msg queue. This is what the application reads when recvmsg() is called. An application can read this anytime after the msg is placed on the queue. The recvmsg hook will also read msg->skb and then after user space reads the msg will call consume_skb(skb) on it effectively free'ing it. But, the race is in above where backlog queue still has a reference to the skb and calls skb_dequeue(). If the skb_dequeue happens after the user reads and free's the skb we have a use after free. The !ingress case does not suffer from this problem because it uses sendmsg_*(sk, msg) which does not pass the sk_buff further down the stack. The following splat was observed with 'test_progs -t sockmap_listen': [ 1022.710250][ T2556] general protection fault, ... [...] [ 1022.712830][ T2556] Workqueue: events sk_psock_backlog [ 1022.713262][ T2556] RIP: 0010:skb_dequeue+0x4c/0x80 [ 1022.713653][ T2556] Code: ... [...] [ 1022.720699][ T2556] Call Trace: [ 1022.720984][ T2556] [ 1022.721254][ T2556] ? die_addr+0x32/0x80^M [ 1022.721589][ T2556] ? exc_general_protection+0x25a/0x4b0 [ 1022.722026][ T2556] ? asm_exc_general_protection+0x22/0x30 [ 1022.722489][ T2556] ? skb_dequeue+0x4c/0x80 [ 1022.722854][ T2556] sk_psock_backlog+0x27a/0x300 [ 1022.723243][ T2556] process_one_work+0x2a7/0x5b0 [ 1022.723633][ T2556] worker_thread+0x4f/0x3a0 [ 1022.723998][ T2556] ? __pfx_worker_thread+0x10/0x10 [ 1022.724386][ T2556] kthread+0xfd/0x130 [ 1022.724709][ T2556] ? __pfx_kthread+0x10/0x10 [ 1022.725066][ T2556] ret_from_fork+0x2d/0x50 [ 1022.725409][ T2556] ? __pfx_kthread+0x10/0x10 [ 1022.725799][ T2556] ret_from_fork_asm+0x1b/0x30 [ 1022.726201][ T2556] To fix we add an skb_get() before passing the skb to be enqueued in the engress queue. This bumps the skb->users refcnt so that consume_skb() and kfree_skb will not immediately free the sk_buff. With this we can be sure the skb is still around when we do the dequeue. Then we just need to decrement the refcnt or free the skb in the backlog case which we do by calling kfree_skb() on the ingress case as well as the sendmsg case. Before locking change from fixes tag we had the sock locked so we couldn't race with user and there was no issue here. Fixes: 799aa7f98d53e ("skmsg: Avoid lock_sock() in sk_psock_backlog()") Reported-by: Jiri Olsa Signed-off-by: John Fastabend Signed-off-by: Daniel Borkmann Tested-by: Xu Kuohai Tested-by: Jiri Olsa Link: https://lore.kernel.org/bpf/20230901202137.214666-1-john.fastabend@gmail.com Signed-off-by: Sasha Levin net/core/skmsg.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) accumulated error probability: 0.00 culprit signature: eb507a9005b343c1475941eb7e424e73ca6caff84b424440cd41fe0af47feac9 parent signature: 86d256ac30b4b29de1b3f6f0dabf23efe95d4887322b76398261e4f72b2ce046 revisions tested: 21, total time: 3h59m15.01057439s (build: 1h42m18.441936557s, test: 2h8m25.265924092s) first good commit: 923877254f002ae87d441382bb1096d9e773d56d bpf, sockmap: Fix skb refcnt race after locking changes recipients (to): ["daniel@iogearbox.net" "john.fastabend@gmail.com" "jolsa@kernel.org" "sashal@kernel.org" "xukuohai@huawei.com"] recipients (cc): []