ci2 starts bisection 2025-01-08 23:56:23.896875603 +0000 UTC m=+459704.513393706 bisecting fixing commit since 91786f140358b1e56efdb0feccb337ce3a59c031 building syzkaller on e104824c06ba54122c1d17b5b26dd21c57e427b6 ensuring issue is reproducible on original commit 91786f140358b1e56efdb0feccb337ce3a59c031 testing commit 91786f140358b1e56efdb0feccb337ce3a59c031 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a5b96b0868c2523fc41d4b86176491f8a18fd8b71edee91c69e337134a23f238 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] check whether we can drop unnecessary instrumentation disabling configs for [HANG LEAK UBSAN BUG KASAN ATOMIC_SLEEP], they are not needed testing commit 91786f140358b1e56efdb0feccb337ce3a59c031 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2e3ea3611f0a8deb1fa29c9254869482700e889262ff34a7ac0fba0ae6867b10 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] the bug reproduces without the instrumentation disabling configs for [HANG LEAK UBSAN BUG KASAN ATOMIC_SLEEP], they are not needed kconfig minimization: base=3706 full=7262 leaves diff=1979 split chunks (needed=false): <1979> split chunk #0 of len 1979 into 5 parts testing without sub-chunk 1/5 disabling configs for [HANG LEAK UBSAN BUG KASAN ATOMIC_SLEEP], they are not needed testing commit 91786f140358b1e56efdb0feccb337ce3a59c031 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: e5d526cf1c8ccd6c1f7b94f1ccdb316797db946b21fa5a98bf933f3e15f3dfd8 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [HANG LEAK UBSAN BUG KASAN ATOMIC_SLEEP], they are not needed testing commit 91786f140358b1e56efdb0feccb337ce3a59c031 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: b69a776909bf70e2fd37e8c80c3086db904285de745ae1bc79571d8453fee2e4 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] the chunk can be dropped testing without sub-chunk 3/5 disabling configs for [HANG LEAK UBSAN BUG KASAN ATOMIC_SLEEP], they are not needed testing commit 91786f140358b1e56efdb0feccb337ce3a59c031 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a2073696f4e274fdd3d61cc2d5f91a81223801a45a03b396ffcf973fa88fc373 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] the chunk can be dropped testing without sub-chunk 4/5 disabling configs for [BUG KASAN ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing commit 91786f140358b1e56efdb0feccb337ce3a59c031 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 69625782508507f5e9ae631741226589a81cebf1c9215e3a913ae5e6f6ec7328 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [HANG LEAK UBSAN BUG KASAN ATOMIC_SLEEP], they are not needed testing commit 91786f140358b1e56efdb0feccb337ce3a59c031 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 727260b901a86b8a350762a169154365835dd119f1ecdc6f126d97a360fbcf31 run #0: ignore: lost connection to test machine run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK false negative chance: 0.000 minimized to 395 configs; suspects: [ARCH_ENABLE_MEMORY_HOTREMOVE ATM BCMA BLK_DEV_ZONED BPF_SYSCALL CARDBUS CFG80211 CFG80211_WEXT CMA CONTIG_ALLOC CRYPTO_842 CRYPTO_LZ4 CRYPTO_LZ4HC CRYPTO_LZO CRYPTO_ZSTD DVB_CORE EXTCON FB FRONTSWAP GPIOLIB HID_ZEROPLUS I2C_MUX IIO IRQ_REMAP LIBNVDIMM MEDIA_ANALOG_TV_SUPPORT MEDIA_CAMERA_SUPPORT MEDIA_CEC_SUPPORT MEDIA_CONTROLLER MEDIA_DIGITAL_TV_SUPPORT MEDIA_RADIO_SUPPORT MEDIA_SDR_SUPPORT MEDIA_SUPPORT MEDIA_TEST_SUPPORT MEDIA_USB_SUPPORT MEMORY_HOTPLUG MEMORY_HOTPLUG_SPARSE MEMORY_HOTREMOVE MFD_VIPERBOARD PARPORT PCCARD PCMCIA PHONET RADIO_ADAPTERS RADIO_SI470X RADIO_SI4713 RC_CORE RFKILL SND SOUND SPI SSB TAP TARGET_CORE TUN USB_AMD5536UDC USB_ATM USB_CONFIGFS USB_CONFIGFS_F_TCM USB_CONFIGFS_F_UAC1 USB_CONFIGFS_F_UAC1_LEGACY USB_CONFIGFS_F_UAC2 USB_CONFIGFS_F_UVC USB_CONFIGFS_MASS_STORAGE USB_CONFIGFS_NCM USB_CONFIGFS_OBEX USB_CONFIGFS_PHONET USB_CONFIGFS_RNDIS USB_CONFIGFS_SERIAL USB_CXACRU USB_CYPRESS_CY7C63 USB_CYTHERM USB_DSBR USB_DUMMY_HCD USB_DWC2 USB_DWC2_HOST USB_DWC2_PCI USB_DWC3 USB_DWC3_GADGET USB_DWC3_OF_SIMPLE USB_DWC3_PCI USB_DWC3_ULPI USB_DYNAMIC_MINORS USB_EG20T USB_EHCI_HCD_PLATFORM USB_EHCI_ROOT_HUB_TT USB_EHSET_TEST_FIXTURE USB_EMI26 USB_EMI62 USB_EPSON2888 USB_EZUSB_FX2 USB_FEW_INIT_RETRIES USB_FOTG210_HCD USB_FOTG210_UDC USB_FTDI_ELAN USB_F_ACM USB_F_ECM USB_F_EEM USB_F_FS USB_F_HID USB_F_MASS_STORAGE USB_F_MIDI USB_F_NCM USB_F_OBEX USB_F_PHONET USB_F_PRINTER USB_F_RNDIS USB_F_SERIAL USB_F_SS_LB USB_F_SUBSET USB_F_TCM USB_F_UAC1 USB_F_UAC1_LEGACY USB_F_UAC2 USB_F_UVC USB_GADGET USB_GADGETFS USB_GADGET_DEBUG_FILES USB_GADGET_DEBUG_FS USB_GL860 USB_GOKU USB_GPIO_VBUS USB_GR_UDC USB_GSPCA USB_GSPCA_BENQ USB_GSPCA_CONEX USB_GSPCA_CPIA1 USB_GSPCA_DTCS033 USB_GSPCA_ETOMS USB_GSPCA_FINEPIX USB_GSPCA_JEILINJ USB_GSPCA_JL2005BCD USB_GSPCA_KINECT USB_GSPCA_KONICA USB_GSPCA_MARS USB_GSPCA_MR97310A USB_GSPCA_NW80X USB_GSPCA_OV519 USB_GSPCA_OV534 USB_GSPCA_OV534_9 USB_GSPCA_PAC207 USB_GSPCA_PAC7302 USB_GSPCA_PAC7311 USB_GSPCA_SE401 USB_GSPCA_SN9C2028 USB_GSPCA_SN9C20X USB_GSPCA_SONIXB USB_GSPCA_SONIXJ USB_GSPCA_SPCA1528 USB_GSPCA_SPCA500 USB_GSPCA_SPCA501 USB_GSPCA_SPCA505 USB_GSPCA_SPCA506 USB_GSPCA_SPCA508 USB_GSPCA_SPCA561 USB_GSPCA_SQ905 USB_GSPCA_SQ905C USB_GSPCA_SQ930X USB_GSPCA_STK014 USB_GSPCA_STK1135 USB_GSPCA_STV0680 USB_GSPCA_SUNPLUS USB_GSPCA_T613 USB_GSPCA_TOPRO USB_GSPCA_TOUPTEK USB_GSPCA_TV8532 USB_GSPCA_VC032X USB_GSPCA_VICAM USB_GSPCA_XIRLINK_CIT USB_GSPCA_ZC3XX USB_HACKRF USB_HCD_BCMA USB_HCD_SSB USB_HSIC_USB3503 USB_HSIC_USB4604 USB_HSO USB_HUB_USB251XB USB_IDMOUSE USB_IOWARRIOR USB_IPHETH USB_ISIGHTFW USB_ISP116X_HCD USB_ISP1301 USB_ISP1760 USB_ISP1760_DUAL_ROLE USB_ISP1760_HCD USB_ISP1761_UDC USB_KAWETH USB_KC2190 USB_KEENE USB_LAN78XX USB_LCD USB_LD USB_LEDS_TRIGGER_USBPORT USB_LED_TRIG USB_LEGOTOWER USB_LIBCOMPOSITE USB_LINK_LAYER_TEST USB_M5602 USB_MA901 USB_MAX3421_HCD USB_MDC800 USB_MICROTEK USB_MR800 USB_MSI2500 USB_MUSB_DUAL_ROLE USB_MUSB_HDRC USB_MV_U3D USB_MV_UDC USB_NET2272 USB_NET2272_DMA USB_NET2280 USB_NET_AX88179_178A USB_NET_AX8817X USB_NET_CDCETHER USB_NET_CDC_EEM USB_NET_CDC_MBIM USB_NET_CDC_NCM USB_NET_CDC_SUBSET USB_NET_CDC_SUBSET_ENABLE USB_NET_CH9200 USB_NET_CX82310_ETH USB_NET_DM9601 USB_NET_GL620A USB_NET_HUAWEI_CDC_NCM USB_NET_INT51X1 USB_NET_KALMIA USB_NET_MCS7830 USB_NET_NET1080 USB_NET_PLUSB USB_NET_QMI_WWAN USB_NET_RNDIS_HOST USB_NET_RNDIS_WLAN USB_NET_SMSC75XX USB_NET_SMSC95XX USB_NET_SR9700 USB_NET_SR9800 USB_NET_ZAURUS USB_OHCI_HCD_PLATFORM USB_OTG USB_OTG_FSM USB_OXU210HP_HCD USB_PEGASUS USB_PULSE8_CEC USB_PWC USB_PWC_INPUT_EVDEV USB_PXA27X USB_R8A66597 USB_R8A66597_HCD USB_RAINSHADOW_CEC USB_RAREMONO USB_RAW_GADGET USB_RTL8150 USB_RTL8152 USB_RTL8153_ECM USB_S2255 USB_SERIAL USB_SERIAL_AIRCABLE USB_SERIAL_ARK3116 USB_SERIAL_BELKIN USB_SERIAL_CH341 USB_SERIAL_CONSOLE USB_SERIAL_CP210X USB_SERIAL_CYBERJACK USB_SERIAL_CYPRESS_M8 USB_SERIAL_DEBUG USB_SERIAL_DIGI_ACCELEPORT USB_SERIAL_EDGEPORT USB_SERIAL_EDGEPORT_TI USB_SERIAL_EMPEG USB_SERIAL_F81232 USB_SERIAL_F8153X USB_SERIAL_FTDI_SIO USB_SERIAL_GARMIN USB_SERIAL_GENERIC USB_SERIAL_IPAQ USB_SERIAL_IPW USB_SERIAL_IR USB_SERIAL_IUU USB_SERIAL_KEYSPAN USB_SERIAL_KEYSPAN_PDA USB_SERIAL_KLSI USB_SERIAL_KOBIL_SCT USB_SERIAL_MCT_U232 USB_SERIAL_METRO USB_SERIAL_MOS7715_PARPORT USB_SERIAL_MOS7720 USB_SERIAL_MOS7840 USB_SERIAL_MXUPORT USB_SERIAL_NAVMAN USB_SERIAL_OMNINET USB_SERIAL_OPTICON USB_SERIAL_OPTION USB_SERIAL_OTI6858 USB_SERIAL_PL2303 USB_SERIAL_QCAUX USB_SERIAL_QT2 USB_SERIAL_QUALCOMM USB_SERIAL_SAFE USB_SERIAL_SIERRAWIRELESS USB_SERIAL_SIMPLE USB_SERIAL_SPCP8X5 USB_SERIAL_SSU100 USB_SERIAL_SYMBOL USB_SERIAL_TI USB_SERIAL_UPD78F0730 USB_SERIAL_VISOR USB_SERIAL_WHITEHEAT USB_SERIAL_WISHBONE USB_SERIAL_WWAN USB_SERIAL_XR USB_SERIAL_XSENS_MT USB_SEVSEG USB_SI470X USB_SI4713 USB_SIERRA_NET USB_SISUSBVGA USB_SL811_CS USB_SL811_HCD USB_SL811_HCD_ISO USB_SNP_CORE USB_SPEEDTOUCH USB_STKWEBCAM USB_STORAGE_ALAUDA USB_STORAGE_CYPRESS_ATACB USB_STORAGE_DATAFAB USB_STORAGE_ENE_UB6250 USB_STORAGE_FREECOM USB_STORAGE_ISD200 USB_STORAGE_JUMPSHOT USB_STORAGE_KARMA USB_STORAGE_ONETOUCH USB_STORAGE_SDDR09 USB_STORAGE_SDDR55 USB_STORAGE_USBAT USB_STV06XX USB_TEST USB_TMC USB_TRANCEVIBRATOR USB_U132_HCD USB_UAS USB_UEAGLEATM USB_ULPI_BUS USB_USBNET USB_USS720 USB_U_AUDIO USB_U_ETHER USB_U_SERIAL USB_VIDEO_CLASS USB_VIDEO_CLASS_INPUT_EVDEV USB_VL600 USB_WDM USB_XHCI_DBGCAP USB_XHCI_PLATFORM USB_XUSBATM USB_YUREX USB_ZR364XX USERFAULTFD USERIO USERMODE_DRIVER USER_RETURN_NOTIFIER U_SERIAL_CONSOLE V4L2_MEM2MEM_DEV V4L_TEST_DRIVERS VALIDATE_FS_PARSER VDPA VETH VFIO VFIO_IOMMU_TYPE1 VFIO_PCI VFIO_PCI_CORE VFIO_PCI_INTX VFIO_PCI_MMAP VFIO_VIRQFD VGASTATE VHOST VHOST_CROSS_ENDIAN_LEGACY VHOST_IOTLB VHOST_NET VHOST_RING VHOST_VDPA VHOST_VSOCK VIDEOBUF2_CORE VIDEOBUF2_DMA_CONTIG VIDEOBUF2_DMA_SG VIDEOBUF2_MEMOPS VIDEOBUF2_V4L2 VIDEOBUF2_VMALLOC VIDEOBUF_GEN VIDEOBUF_VMALLOC VIDEO_AU0828 VIDEO_AU0828_RC VIDEO_AU0828_V4L2 VIDEO_CPIA2 VIDEO_CS53L32A VIDEO_CX231XX VIDEO_CX231XX_ALSA VIDEO_CX231XX_DVB VIDEO_CX231XX_RC VIDEO_CX2341X VIDEO_CX25840 VIDEO_DEV VIDEO_EM28XX VIDEO_EM28XX_ALSA VIDEO_EM28XX_DVB VIDEO_EM28XX_RC VIDEO_EM28XX_V4L2 VIDEO_GO7007 VIDEO_GO7007_LOADER VIDEO_GO7007_USB VIDEO_GO7007_USB_S2250_BOARD VIDEO_HDPVR VIDEO_MSP3400 VIDEO_PVRUSB2 VIDEO_PVRUSB2_DVB VIDEO_PVRUSB2_SYSFS VIDEO_SAA711X VIDEO_STK1160 VIDEO_STK1160_COMMON VIDEO_TM6000 VIDEO_TM6000_ALSA VIDEO_TM6000_DVB VIDEO_TUNER VIDEO_TVEEPROM VIDEO_USBTV VIDEO_V4L2 VIDEO_V4L2_I2C VIDEO_V4L2_SUBDEV_API VIDEO_V4L2_TPG VIDEO_VICODEC VIDEO_VIM2M VIDEO_VIMC VIDEO_VIVID VIDEO_VIVID_CEC VIDEO_WM8775 VIPERBOARD_ADC VIRTIO_BALLOON VIRTIO_DMA_SHARED_BUFFER VIRTIO_MEM VIRTIO_MMIO VIRTIO_MMIO_CMDLINE_DEVICES VIRTIO_PMEM VIRTIO_VDPA VIRTIO_VSOCKETS VIRTIO_VSOCKETS_COMMON VIRT_WIFI VLAN_8021Q VLAN_8021Q_GVRP VLAN_8021Q_MVRP VMAP_PFN VMWARE_VMCI VMXNET3 VP_VDPA VSOCKETS VSOCKETS_DIAG VSOCKETS_LOOPBACK VSOCKMON VT_HW_CONSOLE_BINDING VXFS_FS WANT_DEV_COREDUMP WEXT_CORE WEXT_PRIV WEXT_PROC WIREGUARD WIRELESS WIRELESS_EXT WLAN WLAN_VENDOR_ADMTEK X86_SGX X86_X2APIC X86_X32 XARRAY_MULTI XDP_SOCKETS XDP_SOCKETS_DIAG XFRM_ESPINTCP XFRM_INTERFACE XFRM_IPCOMP XFRM_MIGRATE XFRM_OFFLOAD XFRM_STATISTICS XFRM_SUB_POLICY XFRM_USER_COMPAT XFS_FS XFS_POSIX_ACL XFS_QUOTA XFS_RT XOR_BLOCKS YENTA YENTA_ENE_TUNE YENTA_O2 YENTA_RICOH YENTA_TI YENTA_TOSHIBA ZBUD ZEROPLUS_FF ZONEFS_FS ZPOOL ZRAM ZRAM_DEF_COMP_LZORLE ZSMALLOC ZSTD_COMPRESS ZSWAP ZSWAP_COMPRESSOR_DEFAULT_LZO ZSWAP_DEFAULT_ON ZSWAP_ZPOOL_DEFAULT_ZBUD] disabling configs for [KASAN ATOMIC_SLEEP HANG LEAK UBSAN BUG], they are not needed determining the merge base between 91786f140358b1e56efdb0feccb337ce3a59c031 and eea6e4b4dfb8859446177c32961c96726d0117be 8bb7eca972ad531c9b149c0a51ab43a417385813/Linux 5.15 is a merge base, check if it has the bug testing commit 8bb7eca972ad531c9b149c0a51ab43a417385813 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 45105af018e0307766480b52214e657dafdbde707dc7d6b3295d36b573927d3b all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] testing current HEAD eea6e4b4dfb8859446177c32961c96726d0117be testing commit eea6e4b4dfb8859446177c32961c96726d0117be gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: bf9118604b0f9ef07efad80a27ae6301adc82cf3d6ed8cdc5d90782ee2ed8bd1 run #0: ignore: lost connection to test machine run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK false negative chance: 0.000 # git bisect start eea6e4b4dfb8859446177c32961c96726d0117be 8bb7eca972ad531c9b149c0a51ab43a417385813 Bisecting: 140124 revisions left to test after this (roughly 17 steps) [63342b1dd5f884d97b41b0cd04fae01a69e0a762] Merge tag '6.4-rc-smb3-client-fixes-part2' of git://git.samba.org/sfrench/cifs-2.6 determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 63342b1dd5f884d97b41b0cd04fae01a69e0a762 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9d5ab69e5087d3a85a7c57232a4a961d5a01551150704b92f89b70efc1552ff9 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good 63342b1dd5f884d97b41b0cd04fae01a69e0a762 Bisecting: 70867 revisions left to test after this (roughly 16 steps) [1f440397665f4241346e4cc6d93f8b73880815d1] Merge tag 'docs-6.9' of git://git.lwn.net/linux determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 1f440397665f4241346e4cc6d93f8b73880815d1 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 1f3b85eaf51765da1aa04885a38e2eb9e0398646745f1a0cd007716e758d19c2 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good 1f440397665f4241346e4cc6d93f8b73880815d1 Bisecting: 35320 revisions left to test after this (roughly 15 steps) [b3ce7a30847a54a7f96a35e609303d8afecd460b] Merge tag 'drm-next-2024-07-18' of https://gitlab.freedesktop.org/drm/kernel determine whether the revision contains the guilty commit revision 63342b1dd5f884d97b41b0cd04fae01a69e0a762 crashed and is reachable testing commit b3ce7a30847a54a7f96a35e609303d8afecd460b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 84a4d611041fa229a5a915cb3422c2dd228d16e5c9b0cce87567439dec368851 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good b3ce7a30847a54a7f96a35e609303d8afecd460b Bisecting: 17674 revisions left to test after this (roughly 14 steps) [3f2ac59c0d7b4d9f0e87371662a6ba8273b07818] Merge branch 'fix-caching-of-btf-for-kfuncs-in-the-verifier' determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 3f2ac59c0d7b4d9f0e87371662a6ba8273b07818 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3ce3899c0ed03e05db5256f0c6fab84f4cf06427adcc84a68d4baf364ba327ef all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good 3f2ac59c0d7b4d9f0e87371662a6ba8273b07818 Bisecting: 8156 revisions left to test after this (roughly 13 steps) [fcc79e1714e8c2b8e216dc3149812edd37884eef] Merge tag 'net-next-6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit fcc79e1714e8c2b8e216dc3149812edd37884eef gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: c10bec8194474f91ad15a2e4202e59a96aef0fe5482ea7b1faa44d2483c49242 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good fcc79e1714e8c2b8e216dc3149812edd37884eef Bisecting: 4069 revisions left to test after this (roughly 12 steps) [7eef7e306d3c40a0c5b9ff6adc9b273cc894dbd5] Merge tag 'for-6.13/dm-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 7eef7e306d3c40a0c5b9ff6adc9b273cc894dbd5 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: c19808a112762f4bc074a6a25b9b4e55cbd923b4d4cfba18b688b80b4844d96d all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good 7eef7e306d3c40a0c5b9ff6adc9b273cc894dbd5 Bisecting: 2045 revisions left to test after this (roughly 11 steps) [2eff01ee2881becc9daaa0d53477ec202136b1f4] Merge tag 'char-misc-6.13-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc determine whether the revision contains the guilty commit revision b3ce7a30847a54a7f96a35e609303d8afecd460b crashed and is reachable testing commit 2eff01ee2881becc9daaa0d53477ec202136b1f4 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: da88b409daeefb6cec1991e07ee958689a75b886ac8e23cc15a62be2cda976bd all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good 2eff01ee2881becc9daaa0d53477ec202136b1f4 Bisecting: 1022 revisions left to test after this (roughly 10 steps) [07137e925fa951646325762bda6bd2503dfe64c6] xfs: don't lose solo dquot update transactions determine whether the revision contains the guilty commit revision b3ce7a30847a54a7f96a35e609303d8afecd460b crashed and is reachable testing commit 07137e925fa951646325762bda6bd2503dfe64c6 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: c6fb4cba0bf5fb0f6f620d9d522d92f5e860bcad38224a9bc5283d1a52935b9a all runs: OK false negative chance: 0.000 # git bisect bad 07137e925fa951646325762bda6bd2503dfe64c6 Bisecting: 520 revisions left to test after this (roughly 9 steps) [0cb71708c5816569f8addd5c6f33cb9679e73b5b] Merge tag 'pci-v6.13-fixes-1' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 0cb71708c5816569f8addd5c6f33cb9679e73b5b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 15037e09ad5c0cab78db256e5674df2ff337ed67ff49e1692dfcc492c80db27f all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good 0cb71708c5816569f8addd5c6f33cb9679e73b5b Bisecting: 252 revisions left to test after this (roughly 8 steps) [2b90dcd599b62ccaaa367947037235de8e6bc3a7] Merge tag 'sound-6.13-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound determine whether the revision contains the guilty commit revision 63342b1dd5f884d97b41b0cd04fae01a69e0a762 crashed and is reachable testing commit 2b90dcd599b62ccaaa367947037235de8e6bc3a7 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: f9e5a51edc84171b14d91fc2206d11e64a69f629499c4c58f1e0da483c958f1f all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good 2b90dcd599b62ccaaa367947037235de8e6bc3a7 Bisecting: 134 revisions left to test after this (roughly 7 steps) [7503345ac5f5e82fd9a36d6e6b447c016376403a] Merge tag 'block-6.13-20241207' of git://git.kernel.dk/linux determine whether the revision contains the guilty commit revision 2eff01ee2881becc9daaa0d53477ec202136b1f4 crashed and is reachable testing commit 7503345ac5f5e82fd9a36d6e6b447c016376403a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 7c200b9a7b50c347d971253fe4f7bf58eb19fe62daa26c1c8b9f5d51ed0a0841 run #0: ignore: lost connection to test machine run #1: ignore: lost connection to test machine run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK false negative chance: 0.000 # git bisect bad 7503345ac5f5e82fd9a36d6e6b447c016376403a Bisecting: 57 revisions left to test after this (roughly 6 steps) [f3ddc438a29f78f0642bfcf84407c236a0b2bdc7] Merge tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux determine whether the revision contains the guilty commit revision 7eef7e306d3c40a0c5b9ff6adc9b273cc894dbd5 crashed and is reachable testing commit f3ddc438a29f78f0642bfcf84407c236a0b2bdc7 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 528ceed3449a86e8314b45088bd5d39ca39ca394bff66536b35088148460085b all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good f3ddc438a29f78f0642bfcf84407c236a0b2bdc7 Bisecting: 28 revisions left to test after this (roughly 5 steps) [eae6a075e9537dd69891cf77ca5a88fa8a28b4a1] bpf: Handle BPF_EXIST and BPF_NOEXIST for LPM trie determine whether the revision contains the guilty commit revision 1f440397665f4241346e4cc6d93f8b73880815d1 crashed and is reachable testing commit eae6a075e9537dd69891cf77ca5a88fa8a28b4a1 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2a300770aac384bc461d16f1cee1cec968d5381167f7d0421b32567fcc8a1b55 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good eae6a075e9537dd69891cf77ca5a88fa8a28b4a1 Bisecting: 14 revisions left to test after this (roughly 4 steps) [d64fd5f77758d234daf16683e1d4305ffc6a05c5] Merge tag 'nvme-6.13-2024-12-05' of git://git.infradead.org/nvme into block-6.13 determine whether the revision contains the guilty commit revision 0cb71708c5816569f8addd5c6f33cb9679e73b5b crashed and is reachable testing commit d64fd5f77758d234daf16683e1d4305ffc6a05c5 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 1b2fcc033ef78f0d18d310b3c3e5e29e47383fb0fe6122a6564e5d8829489a84 all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good d64fd5f77758d234daf16683e1d4305ffc6a05c5 Bisecting: 7 revisions left to test after this (roughly 3 steps) [509df676c2d79c985ec2eaa3e3a3bbe557645861] Merge branch 'fixes-for-lpm-trie' determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 509df676c2d79c985ec2eaa3e3a3bbe557645861 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 940f6788f319eba07124cbd4a1ed30fdb7f6c21acf1ee200dd9d0f88ede54d92 all runs: OK false negative chance: 0.000 # git bisect bad 509df676c2d79c985ec2eaa3e3a3bbe557645861 Bisecting: 3 revisions left to test after this (roughly 2 steps) [3d8dc43eb2a3d179809f5fc27c88c93a57ea123d] bpf: Switch to bpf mem allocator for LPM trie determine whether the revision contains the guilty commit revision 3f2ac59c0d7b4d9f0e87371662a6ba8273b07818 crashed and is reachable testing commit 3d8dc43eb2a3d179809f5fc27c88c93a57ea123d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2396c5f71352c623e57a0be466f3e1b89f8c18ba4408dfc514fc7a51e73fed7c all runs: OK false negative chance: 0.000 # git bisect bad 3d8dc43eb2a3d179809f5fc27c88c93a57ea123d Bisecting: 0 revisions left to test after this (roughly 1 step) [27abc7b3fa2e09bbe41e2924d328121546865eda] bpf: Fix exact match conditions in trie_get_next_key() determine whether the revision contains the guilty commit revision b3ce7a30847a54a7f96a35e609303d8afecd460b crashed and is reachable testing commit 27abc7b3fa2e09bbe41e2924d328121546865eda gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ef86b56f0418046fbd77a2a8e0d5b15acd2b765167ad512984d919c2095b0b6f all runs: crashed: possible deadlock in trie_delete_elem representative crash: possible deadlock in trie_delete_elem, types: [LOCKDEP] # git bisect good 27abc7b3fa2e09bbe41e2924d328121546865eda 3d8dc43eb2a3d179809f5fc27c88c93a57ea123d is the first bad commit commit 3d8dc43eb2a3d179809f5fc27c88c93a57ea123d Author: Hou Tao Date: Fri Dec 6 19:06:19 2024 +0800 bpf: Switch to bpf mem allocator for LPM trie Multiple syzbot warnings have been reported. These warnings are mainly about the lock order between trie->lock and kmalloc()'s internal lock. See report [1] as an example: ====================================================== WARNING: possible circular locking dependency detected 6.10.0-rc7-syzkaller-00003-g4376e966ecb7 #0 Not tainted ------------------------------------------------------ syz.3.2069/15008 is trying to acquire lock: ffff88801544e6d8 (&n->list_lock){-.-.}-{2:2}, at: get_partial_node ... but task is already holding lock: ffff88802dcc89f8 (&trie->lock){-.-.}-{2:2}, at: trie_update_elem ... which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #1 (&trie->lock){-.-.}-{2:2}: __raw_spin_lock_irqsave _raw_spin_lock_irqsave+0x3a/0x60 trie_delete_elem+0xb0/0x820 ___bpf_prog_run+0x3e51/0xabd0 __bpf_prog_run32+0xc1/0x100 bpf_dispatcher_nop_func ...... bpf_trace_run2+0x231/0x590 __bpf_trace_contention_end+0xca/0x110 trace_contention_end.constprop.0+0xea/0x170 __pv_queued_spin_lock_slowpath+0x28e/0xcc0 pv_queued_spin_lock_slowpath queued_spin_lock_slowpath queued_spin_lock do_raw_spin_lock+0x210/0x2c0 __raw_spin_lock_irqsave _raw_spin_lock_irqsave+0x42/0x60 __put_partials+0xc3/0x170 qlink_free qlist_free_all+0x4e/0x140 kasan_quarantine_reduce+0x192/0x1e0 __kasan_slab_alloc+0x69/0x90 kasan_slab_alloc slab_post_alloc_hook slab_alloc_node kmem_cache_alloc_node_noprof+0x153/0x310 __alloc_skb+0x2b1/0x380 ...... -> #0 (&n->list_lock){-.-.}-{2:2}: check_prev_add check_prevs_add validate_chain __lock_acquire+0x2478/0x3b30 lock_acquire lock_acquire+0x1b1/0x560 __raw_spin_lock_irqsave _raw_spin_lock_irqsave+0x3a/0x60 get_partial_node.part.0+0x20/0x350 get_partial_node get_partial ___slab_alloc+0x65b/0x1870 __slab_alloc.constprop.0+0x56/0xb0 __slab_alloc_node slab_alloc_node __do_kmalloc_node __kmalloc_node_noprof+0x35c/0x440 kmalloc_node_noprof bpf_map_kmalloc_node+0x98/0x4a0 lpm_trie_node_alloc trie_update_elem+0x1ef/0xe00 bpf_map_update_value+0x2c1/0x6c0 map_update_elem+0x623/0x910 __sys_bpf+0x90c/0x49a0 ... other info that might help us debug this: Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&trie->lock); lock(&n->list_lock); lock(&trie->lock); lock(&n->list_lock); *** DEADLOCK *** [1]: https://syzkaller.appspot.com/bug?extid=9045c0a3d5a7f1b119f7 A bpf program attached to trace_contention_end() triggers after acquiring &n->list_lock. The program invokes trie_delete_elem(), which then acquires trie->lock. However, it is possible that another process is invoking trie_update_elem(). trie_update_elem() will acquire trie->lock first, then invoke kmalloc_node(). kmalloc_node() may invoke get_partial_node() and try to acquire &n->list_lock (not necessarily the same lock object). Therefore, lockdep warns about the circular locking dependency. Invoking kmalloc() before acquiring trie->lock could fix the warning. However, since BPF programs call be invoked from any context (e.g., through kprobe/tracepoint/fentry), there may still be lock ordering problems for internal locks in kmalloc() or trie->lock itself. To eliminate these potential lock ordering problems with kmalloc()'s internal locks, replacing kmalloc()/kfree()/kfree_rcu() with equivalent BPF memory allocator APIs that can be invoked in any context. The lock ordering problems with trie->lock (e.g., reentrance) will be handled separately. Three aspects of this change require explanation: 1. Intermediate and leaf nodes are allocated from the same allocator. Since the value size of LPM trie is usually small, using a single alocator reduces the memory overhead of the BPF memory allocator. 2. Leaf nodes are allocated before disabling IRQs. This handles cases where leaf_size is large (e.g., > 4KB - 8) and updates require intermediate node allocation. If leaf nodes were allocated in IRQ-disabled region, the free objects in BPF memory allocator would not be refilled timely and the intermediate node allocation may fail. 3. Paired migrate_{disable|enable}() calls for node alloc and free. The BPF memory allocator uses per-CPU struct internally, these paired calls are necessary to guarantee correctness. Reviewed-by: Toke Høiland-Jørgensen Signed-off-by: Hou Tao Link: https://lore.kernel.org/r/20241206110622.1161752-7-houtao@huaweicloud.com Signed-off-by: Alexei Starovoitov kernel/bpf/lpm_trie.c | 71 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 48 insertions(+), 23 deletions(-) accumulated error probability: 0.00 culprit signature: 2396c5f71352c623e57a0be466f3e1b89f8c18ba4408dfc514fc7a51e73fed7c parent signature: ef86b56f0418046fbd77a2a8e0d5b15acd2b765167ad512984d919c2095b0b6f revisions tested: 26, total time: 7h14m32.960889491s (build: 4h17m21.329671223s, test: 2h40m55.122972462s) first good commit: 3d8dc43eb2a3d179809f5fc27c88c93a57ea123d bpf: Switch to bpf mem allocator for LPM trie recipients (to): ["ast@kernel.org" "houtao1@huawei.com" "toke@redhat.com"] recipients (cc): []