ci starts bisection 2024-06-16 06:44:55.150298048 +0000 UTC m=+81824.751591955 bisecting cause commit starting from 36534d3c54537bf098224a32dc31397793d4594d building syzkaller on f429ab00b60520c365d122f14c46235ca0aa11ef ensuring issue is reproducible on original commit 36534d3c54537bf098224a32dc31397793d4594d testing commit 36534d3c54537bf098224a32dc31397793d4594d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 48d3f25eb729d935b706649e9e11e4fb5d85008fb417685b9398add65acf851e run #0: crashed: inconsistent lock state in valid_state run #1: crashed: inconsistent lock state in valid_state run #2: crashed: inconsistent lock state in valid_state run #3: crashed: possible deadlock in __mmap_lock_do_trace_start_locking run #4: crashed: inconsistent lock state in valid_state run #5: crashed: inconsistent lock state in valid_state run #6: crashed: inconsistent lock state in valid_state run #7: crashed: inconsistent lock state in valid_state run #8: crashed: inconsistent lock state in valid_state run #9: crashed: possible deadlock in __mmap_lock_do_trace_start_locking run #10: crashed: possible deadlock in __mmap_lock_do_trace_start_locking run #11: crashed: inconsistent lock state in valid_state run #12: crashed: possible deadlock in __mmap_lock_do_trace_start_locking run #13: crashed: inconsistent lock state in valid_state run #14: crashed: inconsistent lock state in valid_state run #15: crashed: possible deadlock in __mmap_lock_do_trace_start_locking run #16: crashed: inconsistent lock state in valid_state run #17: crashed: inconsistent lock state in valid_state run #18: crashed: inconsistent lock state in valid_state run #19: crashed: inconsistent lock state in valid_state representative crash: inconsistent lock state in valid_state, types: [LOCKDEP] check whether we can drop unnecessary instrumentation disabling configs for [LEAK UBSAN BUG KASAN ATOMIC_SLEEP HANG], they are not needed testing commit 36534d3c54537bf098224a32dc31397793d4594d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 4ed20fd08bd2862fb0d817a057cd370fa96efe2904a1ec47246b1b08f5e1b0df all runs: crashed: inconsistent lock state in __mmap_lock_do_trace_released representative crash: inconsistent lock state in __mmap_lock_do_trace_released, types: [LOCKDEP] the bug reproduces without the instrumentation disabling configs for [UBSAN BUG KASAN ATOMIC_SLEEP HANG LEAK], they are not needed kconfig minimization: base=3976 full=8044 leaves diff=2024 split chunks (needed=false): <2024> split chunk #0 of len 2024 into 5 parts testing without sub-chunk 1/5 disabling configs for [HANG LEAK UBSAN BUG KASAN ATOMIC_SLEEP], they are not needed testing commit 36534d3c54537bf098224a32dc31397793d4594d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: c473b11baf08c119d09eb718153711ab671dcc4ffc6551569049e1a1ecf63f1d all runs: crashed: inconsistent lock state in __mmap_lock_do_trace_released representative crash: inconsistent lock state in __mmap_lock_do_trace_released, types: [LOCKDEP] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [BUG KASAN ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing commit 36534d3c54537bf098224a32dc31397793d4594d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: cab36672f8cab8c4fdea56a75febad891536f2f0ac9a602cfda8eff8d05ffb02 all runs: crashed: inconsistent lock state in __mmap_lock_do_trace_released representative crash: inconsistent lock state in __mmap_lock_do_trace_released, types: [LOCKDEP] the chunk can be dropped testing without sub-chunk 3/5 disabling configs for [HANG LEAK UBSAN BUG KASAN ATOMIC_SLEEP], they are not needed testing commit 36534d3c54537bf098224a32dc31397793d4594d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 09c30d082b38120ab5f0cc1facdf4f90fb3a7d795655825761dcb3bc72a4bb72 all runs: crashed: inconsistent lock state in __mmap_lock_do_trace_released representative crash: inconsistent lock state in __mmap_lock_do_trace_released, types: [LOCKDEP] the chunk can be dropped testing without sub-chunk 4/5 disabling configs for [HANG LEAK UBSAN BUG KASAN ATOMIC_SLEEP], they are not needed testing commit 36534d3c54537bf098224a32dc31397793d4594d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ae21769b568d93eec736b104fe9f971fa916102650461e6f9349bba7fbc3da52 all runs: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned representative crash: inconsistent lock state in __mmap_lock_do_trace_acquire_returned, types: [LOCKDEP] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [LEAK UBSAN BUG KASAN ATOMIC_SLEEP HANG], they are not needed testing commit 36534d3c54537bf098224a32dc31397793d4594d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ed13f875c70b23386eedfa8567539ad597a3e15b60568223794fdd4480eed999 all runs: OK false negative chance: 0.000 minimized to 404 configs; suspects: [ARCH_ENABLE_MEMORY_HOTREMOVE ATM BCMA BLK_DEV_ZONED BPF_SYSCALL CARDBUS CFG80211 CFG80211_WEXT CMA COMMON_CLK CONTIG_ALLOC CRYPTO_842 CRYPTO_LZ4 CRYPTO_LZ4HC CRYPTO_ZSTD DVB_CORE EXTCON FB GPIOLIB HID_ZEROPLUS I2C_MUX IIO IOMMUFD IRQ_REMAP KVM KVM_INTEL LIBNVDIMM MEDIA_ANALOG_TV_SUPPORT MEDIA_CAMERA_SUPPORT MEDIA_CEC_SUPPORT MEDIA_CONTROLLER MEDIA_DIGITAL_TV_SUPPORT MEDIA_RADIO_SUPPORT MEDIA_SDR_SUPPORT MEDIA_SUPPORT MEDIA_TEST_SUPPORT MEDIA_USB_SUPPORT MEMORY_HOTPLUG MEMORY_HOTREMOVE MFD_VIPERBOARD PARPORT PCCARD PCMCIA PHONET RADIO_ADAPTERS RADIO_SI470X RADIO_SI4713 RC_CORE RFKILL SND SOUND SPI SSB TAP TARGET_CORE TUN USB_AMD5536UDC USB_ATM USB_CHIPIDEA USB_CHIPIDEA_UDC USB_CONFIGFS USB_CONFIGFS_ACM USB_CONFIGFS_ECM USB_CONFIGFS_ECM_SUBSET USB_CONFIGFS_EEM USB_CONFIGFS_F_FS USB_CONFIGFS_F_HID USB_CONFIGFS_F_LB_SS USB_CONFIGFS_F_MIDI USB_CONFIGFS_F_PRINTER USB_CONFIGFS_F_TCM USB_CONFIGFS_F_UAC1 USB_CONFIGFS_F_UAC1_LEGACY USB_CONFIGFS_F_UAC2 USB_CONFIGFS_F_UVC USB_CONFIGFS_MASS_STORAGE USB_CONFIGFS_NCM USB_CONFIGFS_OBEX USB_CONFIGFS_PHONET USB_CONFIGFS_RNDIS USB_CONFIGFS_SERIAL USB_CXACRU USB_CYPRESS_CY7C63 USB_CYTHERM USB_DSBR USB_DUMMY_HCD USB_DWC2 USB_DWC2_HOST USB_DWC2_PCI USB_DWC3 USB_DWC3_GADGET USB_DWC3_OF_SIMPLE USB_DWC3_PCI USB_DWC3_ULPI USB_DYNAMIC_MINORS USB_EG20T USB_EHCI_HCD_PLATFORM USB_EHCI_ROOT_HUB_TT USB_EHSET_TEST_FIXTURE USB_EMI26 USB_EMI62 USB_EPSON2888 USB_EZUSB_FX2 USB_FEW_INIT_RETRIES USB_F_ACM USB_F_ECM USB_F_EEM USB_F_FS USB_F_HID USB_F_MASS_STORAGE USB_F_MIDI USB_F_NCM USB_F_OBEX USB_F_PHONET USB_F_PRINTER USB_F_RNDIS USB_F_SERIAL USB_F_SS_LB USB_F_SUBSET USB_F_TCM USB_F_UAC1 USB_F_UAC1_LEGACY USB_F_UAC2 USB_F_UVC USB_GADGET USB_GADGETFS USB_GADGET_DEBUG_FILES USB_GADGET_DEBUG_FS USB_GL860 USB_GOKU USB_GPIO_VBUS USB_GR_UDC USB_GSPCA USB_GSPCA_BENQ USB_GSPCA_CONEX USB_GSPCA_CPIA1 USB_GSPCA_DTCS033 USB_GSPCA_ETOMS USB_GSPCA_FINEPIX USB_GSPCA_JEILINJ USB_GSPCA_JL2005BCD USB_GSPCA_KINECT USB_GSPCA_KONICA USB_GSPCA_MARS USB_GSPCA_MR97310A USB_GSPCA_NW80X USB_GSPCA_OV519 USB_GSPCA_OV534 USB_GSPCA_OV534_9 USB_GSPCA_PAC207 USB_GSPCA_PAC7302 USB_GSPCA_PAC7311 USB_GSPCA_SE401 USB_GSPCA_SN9C2028 USB_GSPCA_SN9C20X USB_GSPCA_SONIXB USB_GSPCA_SONIXJ USB_GSPCA_SPCA1528 USB_GSPCA_SPCA500 USB_GSPCA_SPCA501 USB_GSPCA_SPCA505 USB_GSPCA_SPCA506 USB_GSPCA_SPCA508 USB_GSPCA_SPCA561 USB_GSPCA_SQ905 USB_GSPCA_SQ905C USB_GSPCA_SQ930X USB_GSPCA_STK014 USB_GSPCA_STK1135 USB_GSPCA_STV0680 USB_GSPCA_SUNPLUS USB_GSPCA_T613 USB_GSPCA_TOPRO USB_GSPCA_TOUPTEK USB_GSPCA_TV8532 USB_GSPCA_VC032X USB_GSPCA_VICAM USB_GSPCA_XIRLINK_CIT USB_GSPCA_ZC3XX USB_HACKRF USB_HCD_BCMA USB_HCD_SSB USB_HSIC_USB3503 USB_HSIC_USB4604 USB_HSO USB_HUB_USB251XB USB_IDMOUSE USB_IOWARRIOR USB_IPHETH USB_ISIGHTFW USB_ISP116X_HCD USB_ISP1301 USB_ISP1760 USB_ISP1760_DUAL_ROLE USB_ISP1760_HCD USB_ISP1761_UDC USB_KAWETH USB_KC2190 USB_KEENE USB_LAN78XX USB_LCD USB_LD USB_LEDS_TRIGGER_USBPORT USB_LED_TRIG USB_LEGOTOWER USB_LIBCOMPOSITE USB_LINK_LAYER_TEST USB_M5602 USB_MA901 USB_MAX3421_HCD USB_MDC800 USB_MICROTEK USB_MR800 USB_MSI2500 USB_MUSB_DUAL_ROLE USB_MUSB_HDRC USB_MV_U3D USB_MV_UDC USB_NET2272 USB_NET2272_DMA USB_NET2280 USB_NET_AX88179_178A USB_NET_AX8817X USB_NET_CDCETHER USB_NET_CDC_EEM USB_NET_CDC_MBIM USB_NET_CDC_NCM USB_NET_CDC_SUBSET USB_NET_CDC_SUBSET_ENABLE USB_NET_CH9200 USB_NET_CX82310_ETH USB_NET_DM9601 USB_NET_GL620A USB_NET_HUAWEI_CDC_NCM USB_NET_INT51X1 USB_NET_KALMIA USB_NET_MCS7830 USB_NET_NET1080 USB_NET_PLUSB USB_NET_QMI_WWAN USB_NET_RNDIS_HOST USB_NET_SMSC75XX USB_NET_SMSC95XX USB_NET_SR9700 USB_NET_SR9800 USB_NET_ZAURUS USB_OHCI_HCD_PLATFORM USB_OTG USB_OTG_FSM USB_OXU210HP_HCD USB_PEGASUS USB_PULSE8_CEC USB_PWC USB_PWC_INPUT_EVDEV USB_PXA27X USB_R8A66597 USB_R8A66597_HCD USB_RAINSHADOW_CEC USB_RAREMONO USB_RAW_GADGET USB_RTL8150 USB_RTL8152 USB_RTL8153_ECM USB_S2255 USB_SERIAL USB_SERIAL_AIRCABLE USB_SERIAL_ARK3116 USB_SERIAL_BELKIN USB_SERIAL_CH341 USB_SERIAL_CONSOLE USB_SERIAL_CP210X USB_SERIAL_CYBERJACK USB_SERIAL_CYPRESS_M8 USB_SERIAL_DEBUG USB_SERIAL_DIGI_ACCELEPORT USB_SERIAL_EDGEPORT USB_SERIAL_EDGEPORT_TI USB_SERIAL_EMPEG USB_SERIAL_F81232 USB_SERIAL_F8153X USB_SERIAL_FTDI_SIO USB_SERIAL_GARMIN USB_SERIAL_GENERIC USB_SERIAL_IPAQ USB_SERIAL_IPW USB_SERIAL_IR USB_SERIAL_IUU USB_SERIAL_KEYSPAN USB_SERIAL_KEYSPAN_PDA USB_SERIAL_KLSI USB_SERIAL_KOBIL_SCT USB_SERIAL_MCT_U232 USB_SERIAL_METRO USB_SERIAL_MOS7715_PARPORT USB_SERIAL_MOS7720 USB_SERIAL_MOS7840 USB_SERIAL_MXUPORT USB_SERIAL_NAVMAN USB_SERIAL_OMNINET USB_SERIAL_OPTICON USB_SERIAL_OPTION USB_SERIAL_OTI6858 USB_SERIAL_PL2303 USB_SERIAL_QCAUX USB_SERIAL_QT2 USB_SERIAL_QUALCOMM USB_SERIAL_SAFE USB_SERIAL_SIERRAWIRELESS USB_SERIAL_SIMPLE USB_SERIAL_SPCP8X5 USB_SERIAL_SSU100 USB_SERIAL_SYMBOL USB_SERIAL_TI USB_SERIAL_UPD78F0730 USB_SERIAL_VISOR USB_SERIAL_WHITEHEAT USB_SERIAL_WISHBONE USB_SERIAL_WWAN USB_SERIAL_XR USB_SERIAL_XSENS_MT USB_SEVSEG USB_SI470X USB_SI4713 USB_SIERRA_NET USB_SISUSBVGA USB_SL811_CS USB_SL811_HCD USB_SL811_HCD_ISO USB_SNP_CORE USB_SPEEDTOUCH USB_STORAGE_ALAUDA USB_STORAGE_CYPRESS_ATACB USB_STORAGE_DATAFAB USB_STORAGE_ENE_UB6250 USB_STORAGE_FREECOM USB_STORAGE_ISD200 USB_STORAGE_JUMPSHOT USB_STORAGE_KARMA USB_STORAGE_ONETOUCH USB_STORAGE_SDDR09 USB_STORAGE_SDDR55 USB_STORAGE_USBAT USB_STV06XX USB_TEST USB_TMC USB_TRANCEVIBRATOR USB_UAS USB_UEAGLEATM USB_ULPI_BUS USB_USBNET USB_USS720 USB_U_AUDIO USB_U_ETHER USB_U_SERIAL USB_VIDEO_CLASS USB_VIDEO_CLASS_INPUT_EVDEV USB_VL600 USB_WDM USB_XHCI_DBGCAP USB_XHCI_PLATFORM USB_XUSBATM USB_YUREX USERFAULTFD USERIO USERMODE_DRIVER USER_RETURN_NOTIFIER UVC_COMMON U_SERIAL_CONSOLE V4L2_MEM2MEM_DEV V4L_TEST_DRIVERS VALIDATE_FS_PARSER VDPA VDPA_SIM VDPA_SIM_BLOCK VDPA_SIM_NET VDPA_USER VETH VFIO VFIO_DEVICE_CDEV VFIO_PCI VFIO_PCI_CORE VFIO_PCI_INTX VFIO_PCI_MMAP VFIO_VIRQFD VGASTATE VHOST VHOST_CROSS_ENDIAN_LEGACY VHOST_IOTLB VHOST_NET VHOST_RING VHOST_TASK VHOST_VDPA VHOST_VSOCK VIDEO VIDEOBUF2_CORE VIDEOBUF2_DMA_CONTIG VIDEOBUF2_DMA_SG VIDEOBUF2_MEMOPS VIDEOBUF2_V4L2 VIDEOBUF2_VMALLOC VIDEOMODE_HELPERS VIDEO_AU0828 VIDEO_AU0828_RC VIDEO_AU0828_V4L2 VIDEO_CS53L32A VIDEO_CX231XX VIDEO_CX231XX_ALSA VIDEO_CX231XX_DVB VIDEO_CX231XX_RC VIDEO_CX2341X VIDEO_CX25840 VIDEO_DEV VIDEO_EM28XX VIDEO_EM28XX_ALSA VIDEO_EM28XX_DVB VIDEO_EM28XX_RC VIDEO_EM28XX_V4L2 VIDEO_GO7007 VIDEO_GO7007_LOADER VIDEO_GO7007_USB VIDEO_GO7007_USB_S2250_BOARD VIDEO_HDPVR VIDEO_MSP3400 VIDEO_PVRUSB2 VIDEO_PVRUSB2_DVB VIDEO_PVRUSB2_SYSFS VIDEO_SAA711X VIDEO_STK1160 VIDEO_TUNER VIDEO_TVEEPROM VIDEO_USBTV VIDEO_V4L2_I2C VIDEO_V4L2_SUBDEV_API VIDEO_V4L2_TPG VIDEO_VICODEC VIDEO_VIM2M VIDEO_VIMC VIDEO_VIVID VIDEO_VIVID_CEC VIDEO_WM8775 VIPERBOARD_ADC VIRTIO_BALLOON VIRTIO_DMA_SHARED_BUFFER VIRTIO_MEM VIRTIO_MMIO VIRTIO_MMIO_CMDLINE_DEVICES VIRTIO_PMEM VIRTIO_VDPA VIRTIO_VSOCKETS VIRTIO_VSOCKETS_COMMON VIRT_WIFI VLAN_8021Q VLAN_8021Q_GVRP VLAN_8021Q_MVRP VMAP_PFN VMWARE_VMCI VMXNET3 VP_VDPA VSOCKETS VSOCKETS_DIAG VSOCKETS_LOOPBACK VSOCKMON VT_HW_CONSOLE_BINDING VXFS_FS WANT_DEV_COREDUMP WEXT_CORE WEXT_PRIV WEXT_PROC WIREGUARD WIRELESS WIRELESS_EXT WLAN WLAN_VENDOR_ADMTEK WLAN_VENDOR_PURELIFI WLAN_VENDOR_SILABS X86_SGX X86_SGX_KVM X86_USER_SHADOW_STACK X86_X2APIC X86_X32_ABI XDP_SOCKETS XDP_SOCKETS_DIAG XFRM_ESPINTCP XFRM_INTERFACE XFRM_IPCOMP XFRM_MIGRATE XFRM_OFFLOAD XFRM_STATISTICS XFRM_SUB_POLICY XFRM_USER_COMPAT XFS_FS XFS_POSIX_ACL XFS_QUOTA XFS_RT XOR_BLOCKS YENTA YENTA_ENE_TUNE YENTA_O2 YENTA_RICOH YENTA_TI YENTA_TOSHIBA ZEROPLUS_FF ZLIB_DEFLATE ZONEFS_FS ZPOOL ZRAM ZRAM_DEF_COMP_LZORLE ZSMALLOC ZSTD_COMPRESS ZSWAP ZSWAP_COMPRESSOR_DEFAULT_LZO ZSWAP_DEFAULT_ON ZSWAP_SHRINKER_DEFAULT_ON ZSWAP_ZPOOL_DEFAULT_ZSMALLOC] disabling configs for [UBSAN BUG KASAN ATOMIC_SLEEP HANG LEAK], they are not needed picked [v6.9 v6.8 v6.7 v6.5 v6.3 v6.1 v5.19 v5.17 v5.14 v5.11 v5.8 v5.5 v5.2 v4.20 v4.19] out of 32 release tags testing release v6.9 testing commit a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 0bd1a85199504eab6fea4a9cb424be6ea41f4a054d541533cbdbed0b3920429b all runs: OK false negative chance: 0.000 # git bisect start 36534d3c54537bf098224a32dc31397793d4594d a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6 Bisecting: 6272 revisions left to test after this (roughly 13 steps) [db5d28c0bfe566908719bec8e25443aabecbb802] Merge tag 'drm-next-2024-05-15' of https://gitlab.freedesktop.org/drm/kernel testing commit db5d28c0bfe566908719bec8e25443aabecbb802 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3d0970a0269dcb2737a3759be087190d15944c9b4af7329a1ff02d4d0023b032 all runs: OK false negative chance: 0.000 # git bisect good db5d28c0bfe566908719bec8e25443aabecbb802 Bisecting: 3137 revisions left to test after this (roughly 12 steps) [0a07e09085e5cff1c8415ebd2ce9b087cf3acc73] Merge tag 'i2c-for-6.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux testing commit 0a07e09085e5cff1c8415ebd2ce9b087cf3acc73 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 0a3a569d1ab2811299417e6fa5f654697c20b0b2407d5c9a41fdbd3e07f45f44 run #0: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #1: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #2: crashed: possible deadlock in try_to_wake_up run #3: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #4: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #5: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #6: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #7: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #8: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #9: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned representative crash: inconsistent lock state in __mmap_lock_do_trace_acquire_returned, types: [LOCKDEP] # git bisect bad 0a07e09085e5cff1c8415ebd2ce9b087cf3acc73 Bisecting: 1544 revisions left to test after this (roughly 11 steps) [ea5f6ad9ad9645733b72ab53a98e719b460d36a6] Merge tag 'platform-drivers-x86-v6.10-1' of git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86 testing commit ea5f6ad9ad9645733b72ab53a98e719b460d36a6 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6311f4a676b22c60ce04e6777fb16840777455f9c6f9329d7508763ca17f83a1 run #0: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #1: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #2: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #3: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #4: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #5: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #6: crashed: possible deadlock in __mmap_lock_do_trace_acquire_returned run #7: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #8: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #9: OK representative crash: inconsistent lock state in __mmap_lock_do_trace_acquire_returned, types: [LOCKDEP] # git bisect bad ea5f6ad9ad9645733b72ab53a98e719b460d36a6 Bisecting: 845 revisions left to test after this (roughly 10 steps) [2e9250022e9f2c9cde3b98fd26dcad1c2a9aedf3] Merge tag 'cxl-for-6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl testing commit 2e9250022e9f2c9cde3b98fd26dcad1c2a9aedf3 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: b43edd0ac45f32c6863e6ae72c93aa9b32ba51082ef8be13519a548993a4c3fd all runs: OK false negative chance: 0.000 # git bisect good 2e9250022e9f2c9cde3b98fd26dcad1c2a9aedf3 Bisecting: 513 revisions left to test after this (roughly 9 steps) [8771b7f31b7fff91a998e6afdb60650d4bac59a5] media: bcm2835-unicam: Depend on COMMON_CLK testing commit 8771b7f31b7fff91a998e6afdb60650d4bac59a5 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d1f3c12e06c0534a1d49ad68b758e66ba3eacc6c4ec2496d6ab9e8068373e754 all runs: OK false negative chance: 0.000 # git bisect good 8771b7f31b7fff91a998e6afdb60650d4bac59a5 Bisecting: 266 revisions left to test after this (roughly 8 steps) [f4b0c4b508364fde023e4f7b9f23f7e38c663dfe] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm testing commit f4b0c4b508364fde023e4f7b9f23f7e38c663dfe gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: c14c9ef80b8c51adfcb34a6c9f753c51c573fc55d849f944578ea79c60faba2a all runs: OK false negative chance: 0.000 # git bisect good f4b0c4b508364fde023e4f7b9f23f7e38c663dfe Bisecting: 128 revisions left to test after this (roughly 7 steps) [b426433c03a6eb547515edbe74ebb3a90b9979dd] Merge tag 'mtd/for-6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux testing commit b426433c03a6eb547515edbe74ebb3a90b9979dd gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d589158a1d0c9430c56913531aedd9d774dfbce9bf539e61248899bc8cec481f run #0: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #1: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #2: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #3: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #4: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #5: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #6: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #7: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #8: crashed: possible deadlock in __mmap_lock_do_trace_acquire_returned run #9: OK representative crash: inconsistent lock state in __mmap_lock_do_trace_acquire_returned, types: [LOCKDEP] # git bisect bad b426433c03a6eb547515edbe74ebb3a90b9979dd Bisecting: 75 revisions left to test after this (roughly 6 steps) [35eea0defb6e46b26e286066e0e77ac5d53e7fba] mmc: renesas_sdhi: Add compatible string for RZ/G2L family, RZ/G3S, and RZ/V2M SoCs testing commit 35eea0defb6e46b26e286066e0e77ac5d53e7fba gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2a22c663fa81f1c4e47da9f548af062c59183506c4592dfbf7e1ba6969087886 all runs: OK false negative chance: 0.000 # git bisect good 35eea0defb6e46b26e286066e0e77ac5d53e7fba Bisecting: 34 revisions left to test after this (roughly 5 steps) [3c999d1ae3c75991902a1a7dad0cb62c2a3008b4] Merge tag 'wq-for-6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq testing commit 3c999d1ae3c75991902a1a7dad0cb62c2a3008b4 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 4d94aa5a50f2cd860ca6fa20b41bb01f71e03dfd4c7b52a1bd3d13a472f2b97d all runs: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned representative crash: inconsistent lock state in __mmap_lock_do_trace_acquire_returned, types: [LOCKDEP] # git bisect bad 3c999d1ae3c75991902a1a7dad0cb62c2a3008b4 Bisecting: 20 revisions left to test after this (roughly 4 steps) [3309ca6f47f11b5d817ce1e5d8b2f1637b93243e] selftests/cgroup: cpu_hogger init: use {} instead of {NULL} testing commit 3309ca6f47f11b5d817ce1e5d8b2f1637b93243e gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 1de2ba92b8dbfcdc7ece26a42cdcdaa9d95a602d6f163c3c43339c26c513d0c3 all runs: OK false negative chance: 0.000 # git bisect good 3309ca6f47f11b5d817ce1e5d8b2f1637b93243e Bisecting: 10 revisions left to test after this (roughly 3 steps) [d6a7bbdde67227127e5e33fb9500bcc4abc40fb3] workqueue: add function in event of workqueue_activate_work testing commit d6a7bbdde67227127e5e33fb9500bcc4abc40fb3 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 75f49e0583d22b53b1ea4dc1e6a5b7d192f38dc9ae30702f481b520d4ff7797a all runs: OK false negative chance: 0.000 # git bisect good d6a7bbdde67227127e5e33fb9500bcc4abc40fb3 Bisecting: 5 revisions left to test after this (roughly 3 steps) [a2a58909cfb5fd5e9f7bb7d954eec0a32fee3f1f] Merge branch 'for-6.10' into test-merge-for-6.10 testing commit a2a58909cfb5fd5e9f7bb7d954eec0a32fee3f1f gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 0cdba3568d37d7600baefaa2380ad423020f776dcf2a57765eb44bff3fa498e3 all runs: OK false negative chance: 0.000 # git bisect good a2a58909cfb5fd5e9f7bb7d954eec0a32fee3f1f Bisecting: 2 revisions left to test after this (roughly 2 steps) [c1457d9aad5ee2feafcf85aa9a58ab50500159d2] selftests/cgroup: Drop define _GNU_SOURCE testing commit c1457d9aad5ee2feafcf85aa9a58ab50500159d2 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 210387eff8e328ab8a119db612b64e0c5d90eebf74319ccea461388be3037bd0 all runs: OK false negative chance: 0.000 # git bisect good c1457d9aad5ee2feafcf85aa9a58ab50500159d2 Bisecting: 0 revisions left to test after this (roughly 1 step) [de6fef50eaf40789b11841474726fd918a3a84a1] Merge tag 'cgroup-for-6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup testing commit de6fef50eaf40789b11841474726fd918a3a84a1 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 375e5f822fc0c478870887b6e5bf6be2582f0b0606126d7a3ae6517c4a829fbd all runs: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned representative crash: inconsistent lock state in __mmap_lock_do_trace_acquire_returned, types: [LOCKDEP] # git bisect bad de6fef50eaf40789b11841474726fd918a3a84a1 Bisecting: 0 revisions left to test after this (roughly 0 steps) [21c38a3bd4ee3fb7337d013a638302fb5e5f9dc2] cgroup/rstat: add cgroup_rstat_cpu_lock helpers and tracepoints testing commit 21c38a3bd4ee3fb7337d013a638302fb5e5f9dc2 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8a1318c6aa2a64533a3c1fb6b44b6eb4fb53a139c92007b003a3c899f09d2140 run #0: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #1: crashed: possible deadlock in __mmap_lock_do_trace_acquire_returned run #2: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #3: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #4: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #5: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #6: crashed: possible deadlock in __mmap_lock_do_trace_acquire_returned run #7: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #8: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned run #9: crashed: inconsistent lock state in __mmap_lock_do_trace_acquire_returned representative crash: inconsistent lock state in __mmap_lock_do_trace_acquire_returned, types: [LOCKDEP] # git bisect bad 21c38a3bd4ee3fb7337d013a638302fb5e5f9dc2 21c38a3bd4ee3fb7337d013a638302fb5e5f9dc2 is the first bad commit commit 21c38a3bd4ee3fb7337d013a638302fb5e5f9dc2 Author: Jesper Dangaard Brouer Date: Wed May 1 16:04:11 2024 +0200 cgroup/rstat: add cgroup_rstat_cpu_lock helpers and tracepoints This closely resembles helpers added for the global cgroup_rstat_lock in commit fc29e04ae1ad ("cgroup/rstat: add cgroup_rstat_lock helpers and tracepoints"). This is for the per CPU lock cgroup_rstat_cpu_lock. Based on production workloads, we observe the fast-path "update" function cgroup_rstat_updated() is invoked around 3 million times per sec, while the "flush" function cgroup_rstat_flush_locked(), walking each possible CPU, can see periodic spikes of 700 invocations/sec. For this reason, the tracepoints are split into normal and fastpath versions for this per-CPU lock. Making it feasible for production to continuously monitor the non-fastpath tracepoint to detect lock contention issues. The reason for monitoring is that lock disables IRQs which can disturb e.g. softirq processing on the local CPUs involved. When the global cgroup_rstat_lock stops disabling IRQs (e.g converted to a mutex), this per CPU lock becomes the next bottleneck that can introduce latency variations. A practical bpftrace script for monitoring contention latency: bpftrace -e ' tracepoint:cgroup:cgroup_rstat_cpu_lock_contended { @start[tid]=nsecs; @cnt[probe]=count()} tracepoint:cgroup:cgroup_rstat_cpu_locked { if (args->contended) { @wait_ns=hist(nsecs-@start[tid]); delete(@start[tid]);} @cnt[probe]=count()} interval:s:1 {time("%H:%M:%S "); print(@wait_ns); print(@cnt); clear(@cnt);}' Signed-off-by: Jesper Dangaard Brouer Signed-off-by: Tejun Heo include/trace/events/cgroup.h | 56 ++++++++++++++++++++++++++++++---- kernel/cgroup/rstat.c | 70 +++++++++++++++++++++++++++++++++++-------- 2 files changed, 108 insertions(+), 18 deletions(-) accumulated error probability: 0.00 culprit signature: 8a1318c6aa2a64533a3c1fb6b44b6eb4fb53a139c92007b003a3c899f09d2140 parent signature: 210387eff8e328ab8a119db612b64e0c5d90eebf74319ccea461388be3037bd0 revisions tested: 23, total time: 7h19m28.21001873s (build: 3h22m26.814982905s, test: 3h32m19.984152636s) first bad commit: 21c38a3bd4ee3fb7337d013a638302fb5e5f9dc2 cgroup/rstat: add cgroup_rstat_cpu_lock helpers and tracepoints recipients (to): ["hawk@kernel.org" "linux-trace-kernel@vger.kernel.org" "mhiramat@kernel.org" "rostedt@goodmis.org" "tj@kernel.org"] recipients (cc): ["cgroups@vger.kernel.org" "hannes@cmpxchg.org" "linux-kernel@vger.kernel.org" "lizefan.x@bytedance.com" "mathieu.desnoyers@efficios.com" "tj@kernel.org"] crash: inconsistent lock state in __mmap_lock_do_trace_acquire_returned ================================ WARNING: inconsistent lock state 6.9.0-rc1-syzkaller #0 Not tainted -------------------------------- inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. syz-executor.2/3102 [HC1[1]:SC0[0]:HE0:SE1] takes: ffff888237c2c0f8 (lock#7){?.+.}-{2:2}, at: local_lock_acquire include/linux/local_lock_internal.h:29 [inline] ffff888237c2c0f8 (lock#7){?.+.}-{2:2}, at: __mmap_lock_do_trace_acquire_returned+0x3e/0x200 mm/mmap_lock.c:237 {HARDIRQ-ON-W} state was registered at: lock_acquire+0xeb/0x270 kernel/locking/lockdep.c:5754 local_lock_acquire include/linux/local_lock_internal.h:29 [inline] __mmap_lock_do_trace_acquire_returned+0x56/0x200 mm/mmap_lock.c:237 __mmap_lock_trace_acquire_returned include/linux/mmap_lock.h:36 [inline] mmap_write_lock_killable include/linux/mmap_lock.h:125 [inline] do_mprotect_pkey+0x586/0x5c0 mm/mprotect.c:711 __do_sys_mprotect mm/mprotect.c:830 [inline] __se_sys_mprotect mm/mprotect.c:827 [inline] __x64_sys_mprotect+0x1d/0x30 mm/mprotect.c:827 do_syscall_64+0xa8/0x1c0 entry_SYSCALL_64_after_hwframe+0x6d/0x75 irq event stamp: 76148 hardirqs last enabled at (76147): [] consume_obj_stock mm/memcontrol.c:3439 [inline] hardirqs last enabled at (76147): [] obj_cgroup_charge+0x123/0x250 mm/memcontrol.c:3561 hardirqs last disabled at (76148): [] sysvec_call_function_single+0xe/0xc0 arch/x86/kernel/smp.c:266 softirqs last enabled at (76094): [] tcp_close+0x27/0x70 net/ipv4/tcp.c:2933 softirqs last disabled at (76092): [] spin_lock_bh include/linux/spinlock.h:356 [inline] softirqs last disabled at (76092): [] release_sock+0x19/0xb0 net/core/sock.c:3548 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(lock#7); lock(lock#7); *** DEADLOCK *** 3 locks held by syz-executor.2/3102: #0: ffffffff8338aad8 (rcu_node_0){-.-.}-{2:2}, at: rcu_report_exp_cpu_mult+0x1a/0xe0 kernel/rcu/tree_exp.h:238 #1: ffffffff83389f50 (rcu_read_lock){....}-{1:2}, at: trace_call_bpf+0x4b/0x3d0 #2: ffff888111635060 (&mm->mmap_lock){++++}-{3:3}, at: mmap_read_trylock include/linux/mmap_lock.h:165 [inline] #2: ffff888111635060 (&mm->mmap_lock){++++}-{3:3}, at: stack_map_get_build_id_offset+0xb5/0x360 kernel/bpf/stackmap.c:141 stack backtrace: CPU: 0 PID: 3102 Comm: syz-executor.2 Not tainted 6.9.0-rc1-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/07/2024 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0xa3/0x100 lib/dump_stack.c:114 mark_lock_irq+0x49a/0x500 mark_lock+0xe9/0x150 kernel/locking/lockdep.c:4678 mark_usage kernel/locking/lockdep.c:4564 [inline] __lock_acquire+0x49d/0x2490 kernel/locking/lockdep.c:5091 lock_acquire+0xeb/0x270 kernel/locking/lockdep.c:5754 local_lock_acquire include/linux/local_lock_internal.h:29 [inline] __mmap_lock_do_trace_acquire_returned+0x56/0x200 mm/mmap_lock.c:237 __mmap_lock_trace_acquire_returned include/linux/mmap_lock.h:36 [inline] mmap_read_trylock include/linux/mmap_lock.h:166 [inline] stack_map_get_build_id_offset+0x252/0x360 kernel/bpf/stackmap.c:141 __bpf_get_stack+0x1d7/0x240 kernel/bpf/stackmap.c:449 ___bpf_prog_run+0x5f6/0x2280 kernel/bpf/core.c:1997 __bpf_prog_run32+0xbb/0xe0 kernel/bpf/core.c:2236 bpf_dispatcher_nop_func include/linux/bpf.h:1234 [inline] __bpf_prog_run include/linux/filter.h:657 [inline] bpf_prog_run include/linux/filter.h:664 [inline] bpf_prog_run_array include/linux/bpf.h:2073 [inline] trace_call_bpf+0x164/0x3d0 kernel/trace/bpf_trace.c:147 perf_trace_run_bpf_submit+0x3b/0xa0 kernel/events/core.c:10161 perf_trace_contention_end+0xbf/0xf0 include/trace/events/lock.h:122 trace_contention_end include/trace/events/lock.h:122 [inline] __pv_queued_spin_lock_slowpath+0x47c/0x4a0 kernel/locking/qspinlock.c:560 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:584 [inline] queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline] queued_spin_lock include/asm-generic/qspinlock.h:114 [inline] do_raw_spin_lock+0x9e/0xb0 kernel/locking/spinlock_debug.c:116 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:111 [inline] _raw_spin_lock_irqsave+0x6d/0xa0 kernel/locking/spinlock.c:162 rcu_report_exp_cpu_mult+0x1a/0xe0 kernel/rcu/tree_exp.h:238 csd_do_func kernel/smp.c:133 [inline] __flush_smp_call_function_queue+0x2ee/0x5d0 kernel/smp.c:542 __sysvec_call_function_single+0x2f/0xf0 arch/x86/kernel/smp.c:271 instr_sysvec_call_function_single arch/x86/kernel/smp.c:266 [inline] sysvec_call_function_single+0x9e/0xc0 arch/x86/kernel/smp.c:266 asm_sysvec_call_function_single+0x1a/0x20 arch/x86/include/asm/idtentry.h:709 RIP: 0010:obj_cgroup_charge+0x14c/0x250 mm/memcontrol.c:3561 Code: ff 48 c7 44 24 10 00 00 00 00 9c 8f 44 24 10 f7 44 24 10 00 02 00 00 0f 85 cf 00 00 00 41 f7 c5 00 02 00 00 74 01 fb 45 31 ed <45> 84 ff 4c 8b 7c 24 08 0f 85 8e 00 00 00 49 89 ee 4c 89 f8 48 c1 RSP: 0018:ffffc9000554fd28 EFLAGS: 00000246 RAX: 279848b651c42d00 RBX: ffff888237c2c848 RCX: 00000000fffffe00 RDX: 000000004b4706dd RSI: ffffffff83001494 RDI: ffffffff82f29c90 RBP: ffff88811a6840c0 R08: 0000000000000001 R09: ffff8881127a5340 R10: 0000000000000000 R11: ffffffff8221bb90 R12: ffffffff8146b1e9 R13: 0000000000000000 R14: ffff8881127a5340 R15: 0000000000000000 __memcg_slab_pre_alloc_hook+0x28d/0x2b0 mm/slub.c:1919 memcg_slab_pre_alloc_hook mm/slub.c:1940 [inline] slab_pre_alloc_hook mm/slub.c:3751 [inline] slab_alloc_node mm/slub.c:3827 [inline] kmem_cache_alloc_lru+0x1d2/0x370 mm/slub.c:3864 alloc_inode_sb include/linux/fs.h:3089 [inline] sock_alloc_inode+0x20/0xa0 net/socket.c:308 alloc_inode+0x1a/0xb0 fs/inode.c:261 new_inode_pseudo+0xc/0x50 fs/inode.c:1007 sock_alloc net/socket.c:634 [inline] __sock_create+0x90/0x420 net/socket.c:1535 sock_create net/socket.c:1622 [inline] __sys_socket_create net/socket.c:1659 [inline] __sys_socket+0x64/0x170 net/socket.c:1706 __do_sys_socket net/socket.c:1720 [inline] __se_sys_socket net/socket.c:1718 [inline] __x64_sys_socket+0x14/0x20 net/socket.c:1718 do_syscall_64+0xa8/0x1c0 entry_SYSCALL_64_after_hwframe+0x6d/0x75 RIP: 0033:0x7f240e629c27 Code: f0 ff ff 77 06 c3 0f 1f 44 00 00 48 c7 c2 b0 ff ff ff f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 b8 29 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007fff55290a08 EFLAGS: 00000202 ORIG_RAX: 0000000000000029 RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f240e629c27 RDX: 0000000000000006 RSI: 0000000000000001 RDI: 0000000000000002 RBP: 00007fff5529114c R08: 000000000000000a R09: 00007fff55290e47 R10: 0000000000000000 R11: 0000000000000202 R12: 00007f240e72fd00 R13: 000000000000d06d R14: 000000000000d043 R15: 00007f240e731ec0 ---------------- Code disassembly (best guess): 0: ff 48 c7 decl -0x39(%rax) 3: 44 24 10 rex.R and $0x10,%al 6: 00 00 add %al,(%rax) 8: 00 00 add %al,(%rax) a: 9c pushf b: 8f 44 24 10 pop 0x10(%rsp) f: f7 44 24 10 00 02 00 testl $0x200,0x10(%rsp) 16: 00 17: 0f 85 cf 00 00 00 jne 0xec 1d: 41 f7 c5 00 02 00 00 test $0x200,%r13d 24: 74 01 je 0x27 26: fb sti 27: 45 31 ed xor %r13d,%r13d * 2a: 45 84 ff test %r15b,%r15b <-- trapping instruction 2d: 4c 8b 7c 24 08 mov 0x8(%rsp),%r15 32: 0f 85 8e 00 00 00 jne 0xc6 38: 49 89 ee mov %rbp,%r14 3b: 4c 89 f8 mov %r15,%rax 3e: 48 rex.W 3f: c1 .byte 0xc1