ci2 starts bisection 2023-12-18 22:59:40.603175308 +0000 UTC m=+22352.115195244 bisecting fixing commit since 9d6bde853685609a631871d7c12be94fdf8d912e building syzkaller on 4bce1a3e705a8b62de8194bdb28f5eef89c8feec ensuring issue is reproducible on original commit 9d6bde853685609a631871d7c12be94fdf8d912e testing commit 9d6bde853685609a631871d7c12be94fdf8d912e gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 5b7c29f9245ec6af4618ffe284e1583cefe17936b2be96af53617becacce1386 run #0: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #1: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #2: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #3: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #4: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #5: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #6: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #7: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #8: crashed: WARNING in xfs_qm_dqget_cache_insert run #9: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #10: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #11: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #12: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #13: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #14: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #15: crashed: WARNING in xfs_qm_dqget_cache_insert run #16: crashed: WARNING in xfs_qm_dqget_cache_insert run #17: crashed: WARNING in xfs_qm_dqget_cache_insert run #18: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #19: crashed: WARNING in xfs_qm_dqget_cache_insert representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] check whether we can drop unnecessary instrumentation disabling configs for [LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN BUG], they are not needed testing commit 9d6bde853685609a631871d7c12be94fdf8d912e gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3df7c2302ffb62ea996b3995bdcf6d009b6df1e65adc735c819103d0d120d909 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] the bug reproduces without the instrumentation disabling configs for [LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN BUG], they are not needed kconfig minimization: base=3703 full=7272 leaves diff=1986 split chunks (needed=false): <1986> split chunk #0 of len 1986 into 5 parts testing without sub-chunk 1/5 disabling configs for [LEAK UBSAN BUG LOCKDEP ATOMIC_SLEEP HANG], they are not needed testing commit 9d6bde853685609a631871d7c12be94fdf8d912e gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: f780c64a0c250ef301fc1368511b8b86d5959ca187b89e15d61f3014fa7d168c all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [HANG LEAK UBSAN BUG LOCKDEP ATOMIC_SLEEP], they are not needed testing commit 9d6bde853685609a631871d7c12be94fdf8d912e gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 69e768b183b56993f7a4d264b0ef6095cd6b8b823afd497a33044ef658038224 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] the chunk can be dropped testing without sub-chunk 3/5 disabling configs for [BUG LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing commit 9d6bde853685609a631871d7c12be94fdf8d912e gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 171e93cc97a82619ddfe3b3376ac90625472d55ee764ee5b1f4d757219155a15 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] the chunk can be dropped testing without sub-chunk 4/5 disabling configs for [LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN BUG], they are not needed testing commit 9d6bde853685609a631871d7c12be94fdf8d912e gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: b5a96f07e49853d60693f7aa8718374a68172a1daf3ccd8777f6cfcee7d08edb all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [ATOMIC_SLEEP HANG LEAK UBSAN BUG LOCKDEP], they are not needed testing commit 9d6bde853685609a631871d7c12be94fdf8d912e gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6f45aa59a36dd2b90f6336889c0ceda31f95194a8b49815df1da958c2301b4ba all runs: OK false negative chance: 0.000 minimized to 394 configs; suspects: [ARCH_ENABLE_MEMORY_HOTREMOVE ATM BCMA BLK_DEV_ZONED BPF_SYSCALL CARDBUS CFG80211 CFG80211_WEXT CMA CONTIG_ALLOC CRYPTO_842 CRYPTO_LZ4 CRYPTO_LZ4HC CRYPTO_LZO CRYPTO_ZSTD DVB_CORE EXTCON FB FRONTSWAP GPIOLIB HID_ZEROPLUS I2C_MUX IIO IRQ_REMAP LIBNVDIMM MEDIA_ANALOG_TV_SUPPORT MEDIA_CAMERA_SUPPORT MEDIA_CEC_SUPPORT MEDIA_CONTROLLER MEDIA_DIGITAL_TV_SUPPORT MEDIA_RADIO_SUPPORT MEDIA_SDR_SUPPORT MEDIA_SUPPORT MEDIA_TEST_SUPPORT MEDIA_USB_SUPPORT MEMORY_HOTPLUG MEMORY_HOTPLUG_SPARSE MEMORY_HOTREMOVE MFD_VIPERBOARD PARPORT PCCARD PCMCIA PHONET RADIO_ADAPTERS RADIO_SI470X RADIO_SI4713 RC_CORE RFKILL SND SOUND SPI SSB TAP TUN USB_AMD5536UDC USB_ATM USB_CONFIGFS USB_CONFIGFS_F_UAC1 USB_CONFIGFS_F_UAC1_LEGACY USB_CONFIGFS_F_UAC2 USB_CONFIGFS_F_UVC USB_CONFIGFS_MASS_STORAGE USB_CONFIGFS_NCM USB_CONFIGFS_OBEX USB_CONFIGFS_PHONET USB_CONFIGFS_RNDIS USB_CONFIGFS_SERIAL USB_CXACRU USB_CYPRESS_CY7C63 USB_CYTHERM USB_DSBR USB_DUMMY_HCD USB_DWC2 USB_DWC2_HOST USB_DWC2_PCI USB_DWC3 USB_DWC3_GADGET USB_DWC3_OF_SIMPLE USB_DWC3_PCI USB_DWC3_ULPI USB_DYNAMIC_MINORS USB_EG20T USB_EHCI_HCD_PLATFORM USB_EHCI_ROOT_HUB_TT USB_EHSET_TEST_FIXTURE USB_EMI26 USB_EMI62 USB_EPSON2888 USB_EZUSB_FX2 USB_FEW_INIT_RETRIES USB_FOTG210_HCD USB_FOTG210_UDC USB_FTDI_ELAN USB_F_ACM USB_F_ECM USB_F_EEM USB_F_FS USB_F_HID USB_F_MASS_STORAGE USB_F_MIDI USB_F_NCM USB_F_OBEX USB_F_PHONET USB_F_PRINTER USB_F_RNDIS USB_F_SERIAL USB_F_SS_LB USB_F_SUBSET USB_F_TCM USB_F_UAC1 USB_F_UAC1_LEGACY USB_F_UAC2 USB_F_UVC USB_GADGET USB_GADGETFS USB_GADGET_DEBUG_FILES USB_GADGET_DEBUG_FS USB_GL860 USB_GOKU USB_GPIO_VBUS USB_GR_UDC USB_GSPCA USB_GSPCA_BENQ USB_GSPCA_CONEX USB_GSPCA_CPIA1 USB_GSPCA_DTCS033 USB_GSPCA_ETOMS USB_GSPCA_FINEPIX USB_GSPCA_JEILINJ USB_GSPCA_JL2005BCD USB_GSPCA_KINECT USB_GSPCA_KONICA USB_GSPCA_MARS USB_GSPCA_MR97310A USB_GSPCA_NW80X USB_GSPCA_OV519 USB_GSPCA_OV534 USB_GSPCA_OV534_9 USB_GSPCA_PAC207 USB_GSPCA_PAC7302 USB_GSPCA_PAC7311 USB_GSPCA_SE401 USB_GSPCA_SN9C2028 USB_GSPCA_SN9C20X USB_GSPCA_SONIXB USB_GSPCA_SONIXJ USB_GSPCA_SPCA1528 USB_GSPCA_SPCA500 USB_GSPCA_SPCA501 USB_GSPCA_SPCA505 USB_GSPCA_SPCA506 USB_GSPCA_SPCA508 USB_GSPCA_SPCA561 USB_GSPCA_SQ905 USB_GSPCA_SQ905C USB_GSPCA_SQ930X USB_GSPCA_STK014 USB_GSPCA_STK1135 USB_GSPCA_STV0680 USB_GSPCA_SUNPLUS USB_GSPCA_T613 USB_GSPCA_TOPRO USB_GSPCA_TOUPTEK USB_GSPCA_TV8532 USB_GSPCA_VC032X USB_GSPCA_VICAM USB_GSPCA_XIRLINK_CIT USB_GSPCA_ZC3XX USB_HACKRF USB_HCD_BCMA USB_HCD_SSB USB_HSIC_USB3503 USB_HSIC_USB4604 USB_HSO USB_HUB_USB251XB USB_IDMOUSE USB_IOWARRIOR USB_IPHETH USB_ISIGHTFW USB_ISP116X_HCD USB_ISP1301 USB_ISP1760 USB_ISP1760_DUAL_ROLE USB_ISP1760_HCD USB_ISP1761_UDC USB_KAWETH USB_KC2190 USB_KEENE USB_LAN78XX USB_LCD USB_LD USB_LEDS_TRIGGER_USBPORT USB_LED_TRIG USB_LEGOTOWER USB_LIBCOMPOSITE USB_LINK_LAYER_TEST USB_M5602 USB_MA901 USB_MAX3421_HCD USB_MDC800 USB_MICROTEK USB_MR800 USB_MSI2500 USB_MUSB_DUAL_ROLE USB_MUSB_HDRC USB_MV_U3D USB_MV_UDC USB_NET2272 USB_NET2272_DMA USB_NET2280 USB_NET_AX88179_178A USB_NET_AX8817X USB_NET_CDCETHER USB_NET_CDC_EEM USB_NET_CDC_MBIM USB_NET_CDC_NCM USB_NET_CDC_SUBSET USB_NET_CDC_SUBSET_ENABLE USB_NET_CH9200 USB_NET_CX82310_ETH USB_NET_DM9601 USB_NET_GL620A USB_NET_HUAWEI_CDC_NCM USB_NET_INT51X1 USB_NET_KALMIA USB_NET_MCS7830 USB_NET_NET1080 USB_NET_PLUSB USB_NET_QMI_WWAN USB_NET_RNDIS_HOST USB_NET_RNDIS_WLAN USB_NET_SMSC75XX USB_NET_SMSC95XX USB_NET_SR9700 USB_NET_SR9800 USB_NET_ZAURUS USB_OHCI_HCD_PLATFORM USB_OTG USB_OTG_FSM USB_OXU210HP_HCD USB_PEGASUS USB_PULSE8_CEC USB_PWC USB_PWC_INPUT_EVDEV USB_PXA27X USB_R8A66597 USB_R8A66597_HCD USB_RAINSHADOW_CEC USB_RAREMONO USB_RAW_GADGET USB_RTL8150 USB_RTL8152 USB_RTL8153_ECM USB_S2255 USB_SERIAL USB_SERIAL_AIRCABLE USB_SERIAL_ARK3116 USB_SERIAL_BELKIN USB_SERIAL_CH341 USB_SERIAL_CONSOLE USB_SERIAL_CP210X USB_SERIAL_CYBERJACK USB_SERIAL_CYPRESS_M8 USB_SERIAL_DEBUG USB_SERIAL_DIGI_ACCELEPORT USB_SERIAL_EDGEPORT USB_SERIAL_EDGEPORT_TI USB_SERIAL_EMPEG USB_SERIAL_F81232 USB_SERIAL_F8153X USB_SERIAL_FTDI_SIO USB_SERIAL_GARMIN USB_SERIAL_GENERIC USB_SERIAL_IPAQ USB_SERIAL_IPW USB_SERIAL_IR USB_SERIAL_IUU USB_SERIAL_KEYSPAN USB_SERIAL_KEYSPAN_PDA USB_SERIAL_KLSI USB_SERIAL_KOBIL_SCT USB_SERIAL_MCT_U232 USB_SERIAL_METRO USB_SERIAL_MOS7715_PARPORT USB_SERIAL_MOS7720 USB_SERIAL_MOS7840 USB_SERIAL_MXUPORT USB_SERIAL_NAVMAN USB_SERIAL_OMNINET USB_SERIAL_OPTICON USB_SERIAL_OPTION USB_SERIAL_OTI6858 USB_SERIAL_PL2303 USB_SERIAL_QCAUX USB_SERIAL_QT2 USB_SERIAL_QUALCOMM USB_SERIAL_SAFE USB_SERIAL_SIERRAWIRELESS USB_SERIAL_SIMPLE USB_SERIAL_SPCP8X5 USB_SERIAL_SSU100 USB_SERIAL_SYMBOL USB_SERIAL_TI USB_SERIAL_UPD78F0730 USB_SERIAL_VISOR USB_SERIAL_WHITEHEAT USB_SERIAL_WISHBONE USB_SERIAL_WWAN USB_SERIAL_XR USB_SERIAL_XSENS_MT USB_SEVSEG USB_SI470X USB_SI4713 USB_SIERRA_NET USB_SISUSBVGA USB_SL811_CS USB_SL811_HCD USB_SL811_HCD_ISO USB_SNP_CORE USB_SPEEDTOUCH USB_STKWEBCAM USB_STORAGE_ALAUDA USB_STORAGE_CYPRESS_ATACB USB_STORAGE_DATAFAB USB_STORAGE_ENE_UB6250 USB_STORAGE_FREECOM USB_STORAGE_ISD200 USB_STORAGE_JUMPSHOT USB_STORAGE_KARMA USB_STORAGE_ONETOUCH USB_STORAGE_SDDR09 USB_STORAGE_SDDR55 USB_STORAGE_USBAT USB_STV06XX USB_TEST USB_TMC USB_TRANCEVIBRATOR USB_U132_HCD USB_UAS USB_UEAGLEATM USB_ULPI_BUS USB_USBNET USB_USS720 USB_U_AUDIO USB_U_ETHER USB_U_SERIAL USB_VIDEO_CLASS USB_VIDEO_CLASS_INPUT_EVDEV USB_VL600 USB_WDM USB_XHCI_DBGCAP USB_XHCI_PLATFORM USB_XUSBATM USB_YUREX USB_ZR364XX USERFAULTFD USERIO USERMODE_DRIVER USER_RETURN_NOTIFIER U_SERIAL_CONSOLE V4L2_MEM2MEM_DEV V4L_TEST_DRIVERS VALIDATE_FS_PARSER VDPA VETH VFIO VFIO_IOMMU_TYPE1 VFIO_PCI VFIO_PCI_CORE VFIO_PCI_INTX VFIO_PCI_MMAP VFIO_VIRQFD VGASTATE VHOST VHOST_CROSS_ENDIAN_LEGACY VHOST_IOTLB VHOST_NET VHOST_RING VHOST_VDPA VHOST_VSOCK VIDEOBUF2_CORE VIDEOBUF2_DMA_CONTIG VIDEOBUF2_DMA_SG VIDEOBUF2_MEMOPS VIDEOBUF2_V4L2 VIDEOBUF2_VMALLOC VIDEOBUF_GEN VIDEOBUF_VMALLOC VIDEO_AU0828 VIDEO_AU0828_RC VIDEO_AU0828_V4L2 VIDEO_CPIA2 VIDEO_CS53L32A VIDEO_CX231XX VIDEO_CX231XX_ALSA VIDEO_CX231XX_DVB VIDEO_CX231XX_RC VIDEO_CX2341X VIDEO_CX25840 VIDEO_DEV VIDEO_EM28XX VIDEO_EM28XX_ALSA VIDEO_EM28XX_DVB VIDEO_EM28XX_RC VIDEO_EM28XX_V4L2 VIDEO_GO7007 VIDEO_GO7007_LOADER VIDEO_GO7007_USB VIDEO_GO7007_USB_S2250_BOARD VIDEO_HDPVR VIDEO_MSP3400 VIDEO_PVRUSB2 VIDEO_PVRUSB2_DVB VIDEO_PVRUSB2_SYSFS VIDEO_SAA711X VIDEO_STK1160 VIDEO_STK1160_COMMON VIDEO_TM6000 VIDEO_TM6000_ALSA VIDEO_TM6000_DVB VIDEO_TUNER VIDEO_TVEEPROM VIDEO_USBTV VIDEO_V4L2 VIDEO_V4L2_I2C VIDEO_V4L2_SUBDEV_API VIDEO_V4L2_TPG VIDEO_VICODEC VIDEO_VIM2M VIDEO_VIMC VIDEO_VIVID VIDEO_VIVID_CEC VIDEO_WM8775 VIPERBOARD_ADC VIRTIO_BALLOON VIRTIO_DMA_SHARED_BUFFER VIRTIO_MEM VIRTIO_MMIO VIRTIO_MMIO_CMDLINE_DEVICES VIRTIO_PMEM VIRTIO_VDPA VIRTIO_VSOCKETS VIRTIO_VSOCKETS_COMMON VIRT_WIFI VLAN_8021Q VLAN_8021Q_GVRP VLAN_8021Q_MVRP VMAP_PFN VMWARE_VMCI VMXNET3 VP_VDPA VSOCKETS VSOCKETS_DIAG VSOCKETS_LOOPBACK VSOCKMON VT_HW_CONSOLE_BINDING VXFS_FS WANT_DEV_COREDUMP WEXT_CORE WEXT_PRIV WEXT_PROC WIREGUARD WIRELESS WIRELESS_EXT WLAN WLAN_VENDOR_ADMTEK X86_SGX X86_X2APIC X86_X32 XARRAY_MULTI XDP_SOCKETS XDP_SOCKETS_DIAG XFRM_ESPINTCP XFRM_INTERFACE XFRM_IPCOMP XFRM_MIGRATE XFRM_OFFLOAD XFRM_STATISTICS XFRM_SUB_POLICY XFRM_USER_COMPAT XFS_FS XFS_POSIX_ACL XFS_QUOTA XFS_RT XFS_SUPPORT_V4 XOR_BLOCKS YENTA YENTA_ENE_TUNE YENTA_O2 YENTA_RICOH YENTA_TI YENTA_TOSHIBA ZBUD ZEROPLUS_FF ZONEFS_FS ZPOOL ZRAM ZRAM_DEF_COMP_LZORLE ZSMALLOC ZSTD_COMPRESS ZSWAP ZSWAP_COMPRESSOR_DEFAULT_LZO ZSWAP_ZPOOL_DEFAULT_ZBUD] disabling configs for [LEAK UBSAN BUG LOCKDEP ATOMIC_SLEEP HANG], they are not needed determining the merge base between 9d6bde853685609a631871d7c12be94fdf8d912e and 2cf4f94d8e8646803f8fb0facf134b0cd7fb691a 8bb7eca972ad531c9b149c0a51ab43a417385813/Linux 5.15 is a merge base, check if it has the bug testing commit 8bb7eca972ad531c9b149c0a51ab43a417385813 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a12a8a10a93c411792725d188fc2124982aadead45274cdc7d6ad4bc8c284c42 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] testing current HEAD 2cf4f94d8e8646803f8fb0facf134b0cd7fb691a testing commit 2cf4f94d8e8646803f8fb0facf134b0cd7fb691a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ac1e6deba5b16361346ec4bcf0edd97dd29d86d54647470a4aa1d1f47b46f90e all runs: OK false negative chance: 0.000 # git bisect start 2cf4f94d8e8646803f8fb0facf134b0cd7fb691a 8bb7eca972ad531c9b149c0a51ab43a417385813 Bisecting: 95193 revisions left to test after this (roughly 17 steps) [5cb0c51fe366cf96b7911d25db3e678401732246] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/klassert/ipsec-next determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 5cb0c51fe366cf96b7911d25db3e678401732246 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: f92e817ad19d921f6df994e4bc04b2fda1cf3f01fec02c7cac7c4df19af553db all runs: OK false negative chance: 0.000 # git bisect bad 5cb0c51fe366cf96b7911d25db3e678401732246 Bisecting: 47548 revisions left to test after this (roughly 16 steps) [cb7f2d05da8ff973444d7e44b9c48e7c90d63915] Merge branch 'ocelot-selftests' determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit cb7f2d05da8ff973444d7e44b9c48e7c90d63915 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 40f527d53652605f94c927c53109ba9186d1984c2cfa9e9a39e32a63930a91cb all runs: OK false negative chance: 0.000 # git bisect bad cb7f2d05da8ff973444d7e44b9c48e7c90d63915 Bisecting: 23772 revisions left to test after this (roughly 15 steps) [809232619f5b15e31fb3563985e705454f32621f] sched/membarrier: Fix membarrier-rseq fence command missing from query bitmask determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 809232619f5b15e31fb3563985e705454f32621f gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 28a3dab6ed38feda3080a404253a13fb585201ac2ba29685ffc6daf297d479ca all runs: OK false negative chance: 0.000 # git bisect bad 809232619f5b15e31fb3563985e705454f32621f Bisecting: 11831 revisions left to test after this (roughly 14 steps) [cb690f5238d71f543f4ce874aa59237cf53a877c] Merge tag 'for-5.16/drivers-2021-11-09' of git://git.kernel.dk/linux-block determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit cb690f5238d71f543f4ce874aa59237cf53a877c gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 268bde9679a0556693842900a38c6805b1d70ab64765fa564f4ac1834460cb2f run #0: failed: failed to run binary in VM: broken console: Permission denied (publickey) run #1: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #2: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #3: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #4: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #5: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #6: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #7: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #8: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #9: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good cb690f5238d71f543f4ce874aa59237cf53a877c Bisecting: 5894 revisions left to test after this (roughly 13 steps) [8d0749b4f83bf4768ceae45ee6a79e6e7eddfc2a] Merge tag 'drm-next-2022-01-07' of git://anongit.freedesktop.org/drm/drm determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 8d0749b4f83bf4768ceae45ee6a79e6e7eddfc2a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 5cbf9227c42e94a68161272befc1d29b866337de2e9f5634b1887a1937445ceb all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good 8d0749b4f83bf4768ceae45ee6a79e6e7eddfc2a Bisecting: 2933 revisions left to test after this (roughly 12 steps) [1be5bdf8cd5a194d981e65687367b0828c839c37] Merge tag 'kcsan.2022.01.09a' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu determine whether the revision contains the guilty commit revision cb690f5238d71f543f4ce874aa59237cf53a877c crashed and is reachable testing commit 1be5bdf8cd5a194d981e65687367b0828c839c37 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 276c8d07bf715f6f2e31a980a0e626215546e9f8188d87ef836bcc265957a050 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good 1be5bdf8cd5a194d981e65687367b0828c839c37 Bisecting: 1461 revisions left to test after this (roughly 11 steps) [e3084ed48fd6b661fe434da0cb36d7d6706cf27f] Merge tag 'pinctrl-v5.17-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit e3084ed48fd6b661fe434da0cb36d7d6706cf27f gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: eb6a52726f29e49a26f76b7b2ac23ec00f4e536fb8cea6bf952aa380d6fa0aaf all runs: OK false negative chance: 0.000 # git bisect bad e3084ed48fd6b661fe434da0cb36d7d6706cf27f Bisecting: 740 revisions left to test after this (roughly 10 steps) [c288ea679840de4dee2ce6da5d0f139e3774ad86] Merge tag 'gpio-updates-for-v5.17' of git://git.kernel.org/pub/scm/linux/kernel/git/brgl/linux determine whether the revision contains the guilty commit revision cb690f5238d71f543f4ce874aa59237cf53a877c crashed and is reachable testing commit c288ea679840de4dee2ce6da5d0f139e3774ad86 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9ec66e30cce11bf4a641baf8e40c5030ccb7ed8b10565ec183f02baac34b3b93 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good c288ea679840de4dee2ce6da5d0f139e3774ad86 Bisecting: 369 revisions left to test after this (roughly 9 steps) [3e3a138a46906641d95599040f2470a60740c399] Merge tag 'for-linus' of git://git.armlinux.org.uk/~rmk/linux-arm determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 3e3a138a46906641d95599040f2470a60740c399 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2622da3a31f52ef1d1e644ed31a81df761850c15e574196c4d45810cca60fc8d all runs: OK false negative chance: 0.000 # git bisect bad 3e3a138a46906641d95599040f2470a60740c399 Bisecting: 187 revisions left to test after this (roughly 8 steps) [d601e58c5f2901783428bc1181e83ff783592b6b] Merge tag 'for-5.17-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux determine whether the revision contains the guilty commit revision 8d0749b4f83bf4768ceae45ee6a79e6e7eddfc2a crashed and is reachable testing commit d601e58c5f2901783428bc1181e83ff783592b6b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8661d41bf5d77a66f2dd67c851e920c564fda3e87281ab98f6d3fe6376b164f9 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good d601e58c5f2901783428bc1181e83ff783592b6b Bisecting: 84 revisions left to test after this (roughly 7 steps) [3f67eaed57dae339603441cf0c0a74ec77a9fc03] Merge tag 'dlm-5.17' of git://git.kernel.org/pub/scm/linux/kernel/git/teigland/linux-dlm determine whether the revision contains the guilty commit revision cb690f5238d71f543f4ce874aa59237cf53a877c crashed and is reachable testing commit 3f67eaed57dae339603441cf0c0a74ec77a9fc03 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 02cc551e52aa662fc852437d86225a579da6d6217a48418bf9f528b383151b76 all runs: OK false negative chance: 0.000 # git bisect bad 3f67eaed57dae339603441cf0c0a74ec77a9fc03 Bisecting: 54 revisions left to test after this (roughly 6 steps) [6eeaf88fd586f05aaf1d48cb3a139d2a5c6eb055] ext4: don't use the orphan list when migrating an inode determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 6eeaf88fd586f05aaf1d48cb3a139d2a5c6eb055 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 70e43950f3d2aeb69779b3d1c0c1db46aaf8d86c5f0f6a84f4758f9ecc6d687f all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good 6eeaf88fd586f05aaf1d48cb3a139d2a5c6eb055 Bisecting: 27 revisions left to test after this (roughly 5 steps) [21d9ac1a5376d949199398848006f6b14649f533] fs: dlm: use event based wait for pending remove determine whether the revision contains the guilty commit checking the merge base 3906fe9bb7f1a2c8667ae54e967dc8690824f4ea no existing result, test the revision testing commit 3906fe9bb7f1a2c8667ae54e967dc8690824f4ea gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9c18da7a9f2f9d32e31880f8ae9b92f883c887a192179d2aacad373798d41324 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] testing commit 21d9ac1a5376d949199398848006f6b14649f533 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9c18da7a9f2f9d32e31880f8ae9b92f883c887a192179d2aacad373798d41324 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good 21d9ac1a5376d949199398848006f6b14649f533 Bisecting: 13 revisions left to test after this (roughly 4 steps) [7e937bb3cbe1f6b9840a43f879aa6e3f1a5e6537] xfs: warn about inodes with project id of -1 determine whether the revision contains the guilty commit revision 3906fe9bb7f1a2c8667ae54e967dc8690824f4ea crashed and is reachable testing commit 7e937bb3cbe1f6b9840a43f879aa6e3f1a5e6537 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: db65ee6d16dae2f27b8fe93b01ab31feb8c974b8873f4bfe1c9244f3fd36c8f0 all runs: OK false negative chance: 0.000 # git bisect bad 7e937bb3cbe1f6b9840a43f879aa6e3f1a5e6537 Bisecting: 6 revisions left to test after this (roughly 3 steps) [132c460e49649685bf4b02ba43dea59062f797d9] xfs: Fix comments mentioning xfs_ialloc determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 132c460e49649685bf4b02ba43dea59062f797d9 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ded1267a48b124dc89d4246701789470f8760baab13117003d1b97e1bd15d2a0 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good 132c460e49649685bf4b02ba43dea59062f797d9 Bisecting: 3 revisions left to test after this (roughly 2 steps) [219aac5d469f7c0bc8ca299dcdeab70380076d24] xfs: sysfs: use default_groups in kobj_type determine whether the revision contains the guilty commit revision 132c460e49649685bf4b02ba43dea59062f797d9 crashed and is reachable testing commit 219aac5d469f7c0bc8ca299dcdeab70380076d24 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6346933d4ffa0e3602d56da9b777678bbdea7f1476290f53cb3795e029a5fcd8 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good 219aac5d469f7c0bc8ca299dcdeab70380076d24 Bisecting: 1 revision left to test after this (roughly 1 step) [f4901a182d33d05a3b7020e2af97c635f6c47959] xfs: Remove redundant assignment of mp determine whether the revision contains the guilty commit revision 132c460e49649685bf4b02ba43dea59062f797d9 crashed and is reachable testing commit f4901a182d33d05a3b7020e2af97c635f6c47959 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a7c70dac285003d32fbb0c818bd6bc4231ab6805ba37089ee9148f8b9562d11e all runs: OK false negative chance: 0.000 # git bisect bad f4901a182d33d05a3b7020e2af97c635f6c47959 Bisecting: 0 revisions left to test after this (roughly 0 steps) [8dc9384b7d75012856b02ff44c37566a55fc2abf] xfs: reduce kvmalloc overhead for CIL shadow buffers determine whether the revision contains the guilty commit revision 132c460e49649685bf4b02ba43dea59062f797d9 crashed and is reachable testing commit 8dc9384b7d75012856b02ff44c37566a55fc2abf gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a7c70dac285003d32fbb0c818bd6bc4231ab6805ba37089ee9148f8b9562d11e all runs: OK false negative chance: 0.000 # git bisect bad 8dc9384b7d75012856b02ff44c37566a55fc2abf 8dc9384b7d75012856b02ff44c37566a55fc2abf is the first bad commit commit 8dc9384b7d75012856b02ff44c37566a55fc2abf Author: Dave Chinner Date: Tue Jan 4 17:22:18 2022 -0800 xfs: reduce kvmalloc overhead for CIL shadow buffers Oh, let me count the ways that the kvmalloc API sucks dog eggs. The problem is when we are logging lots of large objects, we hit kvmalloc really damn hard with costly order allocations, and behaviour utterly sucks: - 49.73% xlog_cil_commit - 31.62% kvmalloc_node - 29.96% __kmalloc_node - 29.38% kmalloc_large_node - 29.33% __alloc_pages - 24.33% __alloc_pages_slowpath.constprop.0 - 18.35% __alloc_pages_direct_compact - 17.39% try_to_compact_pages - compact_zone_order - 15.26% compact_zone 5.29% __pageblock_pfn_to_page 3.71% PageHuge - 1.44% isolate_migratepages_block 0.71% set_pfnblock_flags_mask 1.11% get_pfnblock_flags_mask - 0.81% get_page_from_freelist - 0.59% _raw_spin_lock_irqsave - do_raw_spin_lock __pv_queued_spin_lock_slowpath - 3.24% try_to_free_pages - 3.14% shrink_node - 2.94% shrink_slab.constprop.0 - 0.89% super_cache_count - 0.66% xfs_fs_nr_cached_objects - 0.65% xfs_reclaim_inodes_count 0.55% xfs_perag_get_tag 0.58% kfree_rcu_shrink_count - 2.09% get_page_from_freelist - 1.03% _raw_spin_lock_irqsave - do_raw_spin_lock __pv_queued_spin_lock_slowpath - 4.88% get_page_from_freelist - 3.66% _raw_spin_lock_irqsave - do_raw_spin_lock __pv_queued_spin_lock_slowpath - 1.63% __vmalloc_node - __vmalloc_node_range - 1.10% __alloc_pages_bulk - 0.93% __alloc_pages - 0.92% get_page_from_freelist - 0.89% rmqueue_bulk - 0.69% _raw_spin_lock - do_raw_spin_lock __pv_queued_spin_lock_slowpath 13.73% memcpy_erms - 2.22% kvfree On this workload, that's almost a dozen CPUs all trying to compact and reclaim memory inside kvmalloc_node at the same time. Yet it is regularly falling back to vmalloc despite all that compaction, page and shrinker reclaim that direct reclaim is doing. Copying all the metadata is taking far less CPU time than allocating the storage! Direct reclaim should be considered extremely harmful. This is a high frequency, high throughput, CPU usage and latency sensitive allocation. We've got memory there, and we're using kvmalloc to allow memory allocation to avoid doing lots of work to try to do contiguous allocations. Except it still does *lots of costly work* that is unnecessary. Worse: the only way to avoid the slowpath page allocation trying to do compaction on costly allocations is to turn off direct reclaim (i.e. remove __GFP_RECLAIM_DIRECT from the gfp flags). Unfortunately, the stupid kvmalloc API then says "oh, this isn't a GFP_KERNEL allocation context, so you only get kmalloc!". This cuts off the vmalloc fallback, and this leads to almost instant OOM problems which ends up in filesystems deadlocks, shutdowns and/or kernel crashes. I want some basic kvmalloc behaviour: - kmalloc for a contiguous range with fail fast semantics - no compaction direct reclaim if the allocation enters the slow path. - run normal vmalloc (i.e. GFP_KERNEL) if kmalloc fails The really, really stupid part about this is these kvmalloc() calls are run under memalloc_nofs task context, so all the allocations are always reduced to GFP_NOFS regardless of the fact that kvmalloc requires GFP_KERNEL to be passed in. IOWs, we're already telling kvmalloc to behave differently to the gfp flags we pass in, but it still won't allow vmalloc to be run with anything other than GFP_KERNEL. So, this patch open codes the kvmalloc() in the commit path to have the above described behaviour. The result is we more than halve the CPU time spend doing kvmalloc() in this path and transaction commits with 64kB objects in them more than doubles. i.e. we get ~5x reduction in CPU usage per costly-sized kvmalloc() invocation and the profile looks like this: - 37.60% xlog_cil_commit 16.01% memcpy_erms - 8.45% __kmalloc - 8.04% kmalloc_order_trace - 8.03% kmalloc_order - 7.93% alloc_pages - 7.90% __alloc_pages - 4.05% __alloc_pages_slowpath.constprop.0 - 2.18% get_page_from_freelist - 1.77% wake_all_kswapds .... - __wake_up_common_lock - 0.94% _raw_spin_lock_irqsave - 3.72% get_page_from_freelist - 2.43% _raw_spin_lock_irqsave - 5.72% vmalloc - 5.72% __vmalloc_node_range - 4.81% __get_vm_area_node.constprop.0 - 3.26% alloc_vmap_area - 2.52% _raw_spin_lock - 1.46% _raw_spin_lock 0.56% __alloc_pages_bulk - 4.66% kvfree - 3.25% vfree - __vfree - 3.23% __vunmap - 1.95% remove_vm_area - 1.06% free_vmap_area_noflush - 0.82% _raw_spin_lock - 0.68% _raw_spin_lock - 0.92% _raw_spin_lock - 1.40% kfree - 1.36% __free_pages - 1.35% __free_pages_ok - 1.02% _raw_spin_lock_irqsave It's worth noting that over 50% of the CPU time spent allocating these shadow buffers is now spent on spinlocks. So the shadow buffer allocation overhead is greatly reduced by getting rid of direct reclaim from kmalloc, and could probably be made even less costly if vmalloc() didn't use global spinlocks to protect it's structures. Signed-off-by: Dave Chinner Reviewed-by: Allison Henderson Reviewed-by: Darrick J. Wong Signed-off-by: Darrick J. Wong fs/xfs/xfs_log_cil.c | 46 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 11 deletions(-) accumulated error probability: 0.00 culprit signature: a7c70dac285003d32fbb0c818bd6bc4231ab6805ba37089ee9148f8b9562d11e parent signature: 6346933d4ffa0e3602d56da9b777678bbdea7f1476290f53cb3795e029a5fcd8 revisions tested: 28, total time: 5h34m32.519930125s (build: 1h58m1.546984524s, test: 3h22m40.473464758s) first good commit: 8dc9384b7d75012856b02ff44c37566a55fc2abf xfs: reduce kvmalloc overhead for CIL shadow buffers recipients (to): ["allison.henderson@oracle.com" "dchinner@redhat.com" "djwong@kernel.org"] recipients (cc): []