ci2 starts bisection 2023-10-24 00:22:04.590213096 +0000 UTC m=+105.141095301 bisecting fixing commit since 8a7f2a5c5aa1648edb4f2029c6ec33870afb7a95 building syzkaller on 7939252e4ddf50bbb9912069a40d32f6c83c4f8e ensuring issue is reproducible on original commit 8a7f2a5c5aa1648edb4f2029c6ec33870afb7a95 testing commit 8a7f2a5c5aa1648edb4f2029c6ec33870afb7a95 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 0d0361806a289685f359d0950642d59f765bd0369a4e583e48f20a6d28e5ae78 run #0: infra problem: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS ErrorDetails:[0xc003b30820 0xc003b30910 0xc003b309b0] Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #1: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #2: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #3: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #4: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #5: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #6: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #7: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #8: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #9: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #10: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #11: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #12: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #13: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #14: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #15: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #16: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #17: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #18: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #19: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] check whether we can drop unnecessary instrumentation disabling configs for [BUG LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing commit 8a7f2a5c5aa1648edb4f2029c6ec33870afb7a95 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: f25a2f4df8b6eb2a27aa31ccc627bd725a86c56bf82cfd4007639f86d9c383db all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] the bug reproduces without the instrumentation disabling configs for [ATOMIC_SLEEP HANG LEAK UBSAN BUG LOCKDEP], they are not needed kconfig minimization: base=3703 full=7270 leaves diff=1979 split chunks (needed=false): <1979> split chunk #0 of len 1979 into 5 parts testing without sub-chunk 1/5 disabling configs for [LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN BUG], they are not needed testing commit 8a7f2a5c5aa1648edb4f2029c6ec33870afb7a95 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 52d2536aea01ad56e994f685cd6f3cab281959d5fb2546eeea0c23156166fe14 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [BUG LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing commit 8a7f2a5c5aa1648edb4f2029c6ec33870afb7a95 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8448fc26f3631c9ff0e7c6981461b02855dc22ba262d77fd831c91bc7af2f189 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] the chunk can be dropped testing without sub-chunk 3/5 disabling configs for [LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN BUG], they are not needed testing commit 8a7f2a5c5aa1648edb4f2029c6ec33870afb7a95 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 5ab9b6c562d2e90dcdf1c02deb13f3eff9e403ece7037c665991a34a04bbf048 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] the chunk can be dropped testing without sub-chunk 4/5 disabling configs for [HANG LEAK UBSAN BUG LOCKDEP ATOMIC_SLEEP], they are not needed testing commit 8a7f2a5c5aa1648edb4f2029c6ec33870afb7a95 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d4166c6349e4bdc3046c8ffc1ef4d13774feaba4d6e08ce65c2e63464fede2b6 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [BUG LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing commit 8a7f2a5c5aa1648edb4f2029c6ec33870afb7a95 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 67cbfc55b7fa932cf95c5df760196d75a2a82883454a1081dffbb3b476b12aaa all runs: OK false negative chance: 0.000 minimized to 395 configs; suspects: [ARCH_ENABLE_MEMORY_HOTREMOVE ATM BCMA BLK_DEV_ZONED BPF_SYSCALL CARDBUS CFG80211 CFG80211_WEXT CMA CONTIG_ALLOC CRYPTO_842 CRYPTO_LZ4 CRYPTO_LZ4HC CRYPTO_LZO CRYPTO_ZSTD DVB_CORE EXTCON FB FRONTSWAP GPIOLIB HID_ZEROPLUS I2C_MUX IIO LIBNVDIMM MEDIA_ANALOG_TV_SUPPORT MEDIA_CAMERA_SUPPORT MEDIA_CEC_SUPPORT MEDIA_CONTROLLER MEDIA_DIGITAL_TV_SUPPORT MEDIA_RADIO_SUPPORT MEDIA_SDR_SUPPORT MEDIA_SUPPORT MEDIA_TEST_SUPPORT MEDIA_USB_SUPPORT MEMORY_HOTPLUG MEMORY_HOTPLUG_SPARSE MEMORY_HOTREMOVE MFD_VIPERBOARD PARPORT PCCARD PCMCIA PHONET RADIO_ADAPTERS RADIO_SI470X RADIO_SI4713 RC_CORE RFKILL SND SOUND SPI SSB TAP TARGET_CORE TUN USB_AMD5536UDC USB_ATM USB_CONFIGFS USB_CONFIGFS_F_TCM USB_CONFIGFS_F_UAC1 USB_CONFIGFS_F_UAC1_LEGACY USB_CONFIGFS_F_UAC2 USB_CONFIGFS_F_UVC USB_CONFIGFS_MASS_STORAGE USB_CONFIGFS_NCM USB_CONFIGFS_OBEX USB_CONFIGFS_PHONET USB_CONFIGFS_RNDIS USB_CONFIGFS_SERIAL USB_CXACRU USB_CYPRESS_CY7C63 USB_CYTHERM USB_DSBR USB_DUMMY_HCD USB_DWC2 USB_DWC2_HOST USB_DWC2_PCI USB_DWC3 USB_DWC3_GADGET USB_DWC3_OF_SIMPLE USB_DWC3_PCI USB_DWC3_ULPI USB_DYNAMIC_MINORS USB_EG20T USB_EHCI_HCD_PLATFORM USB_EHCI_ROOT_HUB_TT USB_EHSET_TEST_FIXTURE USB_EMI26 USB_EMI62 USB_EPSON2888 USB_EZUSB_FX2 USB_FEW_INIT_RETRIES USB_FOTG210_HCD USB_FOTG210_UDC USB_FTDI_ELAN USB_F_ACM USB_F_ECM USB_F_EEM USB_F_FS USB_F_HID USB_F_MASS_STORAGE USB_F_MIDI USB_F_NCM USB_F_OBEX USB_F_PHONET USB_F_PRINTER USB_F_RNDIS USB_F_SERIAL USB_F_SS_LB USB_F_SUBSET USB_F_TCM USB_F_UAC1 USB_F_UAC1_LEGACY USB_F_UAC2 USB_F_UVC USB_GADGET USB_GADGETFS USB_GADGET_DEBUG_FILES USB_GADGET_DEBUG_FS USB_GL860 USB_GOKU USB_GPIO_VBUS USB_GR_UDC USB_GSPCA USB_GSPCA_BENQ USB_GSPCA_CONEX USB_GSPCA_CPIA1 USB_GSPCA_DTCS033 USB_GSPCA_ETOMS USB_GSPCA_FINEPIX USB_GSPCA_JEILINJ USB_GSPCA_JL2005BCD USB_GSPCA_KINECT USB_GSPCA_KONICA USB_GSPCA_MARS USB_GSPCA_MR97310A USB_GSPCA_NW80X USB_GSPCA_OV519 USB_GSPCA_OV534 USB_GSPCA_OV534_9 USB_GSPCA_PAC207 USB_GSPCA_PAC7302 USB_GSPCA_PAC7311 USB_GSPCA_SE401 USB_GSPCA_SN9C2028 USB_GSPCA_SN9C20X USB_GSPCA_SONIXB USB_GSPCA_SONIXJ USB_GSPCA_SPCA1528 USB_GSPCA_SPCA500 USB_GSPCA_SPCA501 USB_GSPCA_SPCA505 USB_GSPCA_SPCA506 USB_GSPCA_SPCA508 USB_GSPCA_SPCA561 USB_GSPCA_SQ905 USB_GSPCA_SQ905C USB_GSPCA_SQ930X USB_GSPCA_STK014 USB_GSPCA_STK1135 USB_GSPCA_STV0680 USB_GSPCA_SUNPLUS USB_GSPCA_T613 USB_GSPCA_TOPRO USB_GSPCA_TOUPTEK USB_GSPCA_TV8532 USB_GSPCA_VC032X USB_GSPCA_VICAM USB_GSPCA_XIRLINK_CIT USB_GSPCA_ZC3XX USB_HACKRF USB_HCD_BCMA USB_HCD_SSB USB_HSIC_USB3503 USB_HSIC_USB4604 USB_HSO USB_HUB_USB251XB USB_IDMOUSE USB_IOWARRIOR USB_IPHETH USB_ISIGHTFW USB_ISP116X_HCD USB_ISP1301 USB_ISP1760 USB_ISP1760_DUAL_ROLE USB_ISP1760_HCD USB_ISP1761_UDC USB_KAWETH USB_KC2190 USB_KEENE USB_LAN78XX USB_LCD USB_LD USB_LEDS_TRIGGER_USBPORT USB_LED_TRIG USB_LEGOTOWER USB_LIBCOMPOSITE USB_LINK_LAYER_TEST USB_M5602 USB_MA901 USB_MAX3421_HCD USB_MDC800 USB_MICROTEK USB_MR800 USB_MSI2500 USB_MUSB_DUAL_ROLE USB_MUSB_HDRC USB_MV_U3D USB_MV_UDC USB_NET2272 USB_NET2272_DMA USB_NET2280 USB_NET_AX88179_178A USB_NET_AX8817X USB_NET_CDCETHER USB_NET_CDC_EEM USB_NET_CDC_MBIM USB_NET_CDC_NCM USB_NET_CDC_SUBSET USB_NET_CDC_SUBSET_ENABLE USB_NET_CH9200 USB_NET_CX82310_ETH USB_NET_DM9601 USB_NET_GL620A USB_NET_HUAWEI_CDC_NCM USB_NET_INT51X1 USB_NET_KALMIA USB_NET_MCS7830 USB_NET_NET1080 USB_NET_PLUSB USB_NET_QMI_WWAN USB_NET_RNDIS_HOST USB_NET_RNDIS_WLAN USB_NET_SMSC75XX USB_NET_SMSC95XX USB_NET_SR9700 USB_NET_SR9800 USB_NET_ZAURUS USB_OHCI_HCD_PLATFORM USB_OTG USB_OTG_FSM USB_OXU210HP_HCD USB_PEGASUS USB_PULSE8_CEC USB_PWC USB_PWC_INPUT_EVDEV USB_PXA27X USB_R8A66597 USB_R8A66597_HCD USB_RAINSHADOW_CEC USB_RAREMONO USB_RAW_GADGET USB_RTL8150 USB_RTL8152 USB_RTL8153_ECM USB_S2255 USB_SERIAL USB_SERIAL_AIRCABLE USB_SERIAL_ARK3116 USB_SERIAL_BELKIN USB_SERIAL_CH341 USB_SERIAL_CONSOLE USB_SERIAL_CP210X USB_SERIAL_CYBERJACK USB_SERIAL_CYPRESS_M8 USB_SERIAL_DEBUG USB_SERIAL_DIGI_ACCELEPORT USB_SERIAL_EDGEPORT USB_SERIAL_EDGEPORT_TI USB_SERIAL_EMPEG USB_SERIAL_F81232 USB_SERIAL_F8153X USB_SERIAL_FTDI_SIO USB_SERIAL_GARMIN USB_SERIAL_GENERIC USB_SERIAL_IPAQ USB_SERIAL_IPW USB_SERIAL_IR USB_SERIAL_IUU USB_SERIAL_KEYSPAN USB_SERIAL_KEYSPAN_PDA USB_SERIAL_KLSI USB_SERIAL_KOBIL_SCT USB_SERIAL_MCT_U232 USB_SERIAL_METRO USB_SERIAL_MOS7715_PARPORT USB_SERIAL_MOS7720 USB_SERIAL_MOS7840 USB_SERIAL_MXUPORT USB_SERIAL_NAVMAN USB_SERIAL_OMNINET USB_SERIAL_OPTICON USB_SERIAL_OPTION USB_SERIAL_OTI6858 USB_SERIAL_PL2303 USB_SERIAL_QCAUX USB_SERIAL_QT2 USB_SERIAL_QUALCOMM USB_SERIAL_SAFE USB_SERIAL_SIERRAWIRELESS USB_SERIAL_SIMPLE USB_SERIAL_SPCP8X5 USB_SERIAL_SSU100 USB_SERIAL_SYMBOL USB_SERIAL_TI USB_SERIAL_UPD78F0730 USB_SERIAL_VISOR USB_SERIAL_WHITEHEAT USB_SERIAL_WISHBONE USB_SERIAL_WWAN USB_SERIAL_XR USB_SERIAL_XSENS_MT USB_SEVSEG USB_SI470X USB_SI4713 USB_SIERRA_NET USB_SISUSBVGA USB_SL811_CS USB_SL811_HCD USB_SL811_HCD_ISO USB_SNP_CORE USB_SPEEDTOUCH USB_STKWEBCAM USB_STORAGE_ALAUDA USB_STORAGE_CYPRESS_ATACB USB_STORAGE_DATAFAB USB_STORAGE_ENE_UB6250 USB_STORAGE_FREECOM USB_STORAGE_ISD200 USB_STORAGE_JUMPSHOT USB_STORAGE_KARMA USB_STORAGE_ONETOUCH USB_STORAGE_SDDR09 USB_STORAGE_SDDR55 USB_STORAGE_USBAT USB_STV06XX USB_TEST USB_TMC USB_TRANCEVIBRATOR USB_U132_HCD USB_UAS USB_UEAGLEATM USB_ULPI_BUS USB_USBNET USB_USS720 USB_U_AUDIO USB_U_ETHER USB_U_SERIAL USB_VIDEO_CLASS USB_VIDEO_CLASS_INPUT_EVDEV USB_VL600 USB_WDM USB_XHCI_DBGCAP USB_XHCI_PLATFORM USB_XUSBATM USB_YUREX USB_ZR364XX USERFAULTFD USERIO USERMODE_DRIVER USER_RETURN_NOTIFIER U_SERIAL_CONSOLE V4L2_MEM2MEM_DEV V4L_TEST_DRIVERS VALIDATE_FS_PARSER VDPA VETH VFIO VFIO_IOMMU_TYPE1 VFIO_PCI VFIO_PCI_CORE VFIO_PCI_INTX VFIO_PCI_MMAP VFIO_VIRQFD VGASTATE VHOST VHOST_CROSS_ENDIAN_LEGACY VHOST_IOTLB VHOST_NET VHOST_RING VHOST_VDPA VHOST_VSOCK VIDEOBUF2_CORE VIDEOBUF2_DMA_CONTIG VIDEOBUF2_DMA_SG VIDEOBUF2_MEMOPS VIDEOBUF2_V4L2 VIDEOBUF2_VMALLOC VIDEOBUF_GEN VIDEOBUF_VMALLOC VIDEO_AU0828 VIDEO_AU0828_RC VIDEO_AU0828_V4L2 VIDEO_CPIA2 VIDEO_CS53L32A VIDEO_CX231XX VIDEO_CX231XX_ALSA VIDEO_CX231XX_DVB VIDEO_CX231XX_RC VIDEO_CX2341X VIDEO_CX25840 VIDEO_DEV VIDEO_EM28XX VIDEO_EM28XX_ALSA VIDEO_EM28XX_DVB VIDEO_EM28XX_RC VIDEO_EM28XX_V4L2 VIDEO_GO7007 VIDEO_GO7007_LOADER VIDEO_GO7007_USB VIDEO_GO7007_USB_S2250_BOARD VIDEO_HDPVR VIDEO_MSP3400 VIDEO_PVRUSB2 VIDEO_PVRUSB2_DVB VIDEO_PVRUSB2_SYSFS VIDEO_SAA711X VIDEO_STK1160 VIDEO_STK1160_COMMON VIDEO_TM6000 VIDEO_TM6000_ALSA VIDEO_TM6000_DVB VIDEO_TUNER VIDEO_TVEEPROM VIDEO_USBTV VIDEO_V4L2 VIDEO_V4L2_I2C VIDEO_V4L2_SUBDEV_API VIDEO_V4L2_TPG VIDEO_VICODEC VIDEO_VIM2M VIDEO_VIMC VIDEO_VIVID VIDEO_VIVID_CEC VIDEO_WM8775 VIPERBOARD_ADC VIRTIO_BALLOON VIRTIO_DMA_SHARED_BUFFER VIRTIO_MEM VIRTIO_MMIO VIRTIO_MMIO_CMDLINE_DEVICES VIRTIO_PMEM VIRTIO_VDPA VIRTIO_VSOCKETS VIRTIO_VSOCKETS_COMMON VIRT_WIFI VLAN_8021Q VLAN_8021Q_GVRP VLAN_8021Q_MVRP VMAP_PFN VMWARE_VMCI VMXNET3 VP_VDPA VSOCKETS VSOCKETS_DIAG VSOCKETS_LOOPBACK VSOCKMON VT_HW_CONSOLE_BINDING VXFS_FS WANT_DEV_COREDUMP WEXT_CORE WEXT_PRIV WEXT_PROC WIREGUARD WIRELESS WIRELESS_EXT WLAN WLAN_VENDOR_ADMTEK X86_SGX X86_X2APIC X86_X32 XARRAY_MULTI XDP_SOCKETS XDP_SOCKETS_DIAG XFRM_ESPINTCP XFRM_INTERFACE XFRM_IPCOMP XFRM_MIGRATE XFRM_OFFLOAD XFRM_STATISTICS XFRM_SUB_POLICY XFRM_USER_COMPAT XFS_FS XFS_POSIX_ACL XFS_QUOTA XFS_RT XFS_SUPPORT_V4 XOR_BLOCKS YENTA YENTA_ENE_TUNE YENTA_O2 YENTA_RICOH YENTA_TI YENTA_TOSHIBA ZBUD ZEROPLUS_FF ZONEFS_FS ZPOOL ZRAM ZRAM_DEF_COMP_LZORLE ZSMALLOC ZSTD_COMPRESS ZSWAP ZSWAP_COMPRESSOR_DEFAULT_LZO ZSWAP_ZPOOL_DEFAULT_ZBUD] disabling configs for [BUG LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed determining the merge base between 8a7f2a5c5aa1648edb4f2029c6ec33870afb7a95 and e017769f4ce20dc0d3fa3220d4d359dcc4431274 8bb7eca972ad531c9b149c0a51ab43a417385813/Linux 5.15 is a merge base, check if it has the bug testing commit 8bb7eca972ad531c9b149c0a51ab43a417385813 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: b955164e831b7d2617c1722ac639ef7d38388d82d48c29d45b1263ccd22c55b6 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] testing current HEAD e017769f4ce20dc0d3fa3220d4d359dcc4431274 testing commit e017769f4ce20dc0d3fa3220d4d359dcc4431274 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: abc8866c439db00fc1d7dd9ec358fe5596b5ef94e084aa084612f40fcc37a82b run #0: infra problem: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS ErrorDetails:[0xc0047ff9f0 0xc0047ffae0 0xc0047ffb80] Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK false negative chance: 0.000 # git bisect start e017769f4ce20dc0d3fa3220d4d359dcc4431274 8bb7eca972ad531c9b149c0a51ab43a417385813 Bisecting: 86155 revisions left to test after this (roughly 16 steps) [e08466a7c00733a501d3c5328d29ec974478d717] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit e08466a7c00733a501d3c5328d29ec974478d717 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: bcda742664e40fd51b32199a8519df2411682d7fa729dfef5bc6563f2e8f5efa all runs: OK false negative chance: 0.000 # git bisect bad e08466a7c00733a501d3c5328d29ec974478d717 Bisecting: 42997 revisions left to test after this (roughly 15 steps) [74164d284b2909de0ba13518cc063e9ea9334749] Merge tag 'pwm/for-5.18-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/thierry.reding/linux-pwm determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 74164d284b2909de0ba13518cc063e9ea9334749 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: eb6a390a453d0d7f308b3f60fc5b9ea4d7b5dad0ab9afde4fbaa5c1512c922ea all runs: OK false negative chance: 0.000 # git bisect bad 74164d284b2909de0ba13518cc063e9ea9334749 Bisecting: 21500 revisions left to test after this (roughly 14 steps) [c288ea679840de4dee2ce6da5d0f139e3774ad86] Merge tag 'gpio-updates-for-v5.17' of git://git.kernel.org/pub/scm/linux/kernel/git/brgl/linux determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit c288ea679840de4dee2ce6da5d0f139e3774ad86 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d52017f2e398b47024d3ac2789d49fa740d16878f8f0eb0474755491dffbe514 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good c288ea679840de4dee2ce6da5d0f139e3774ad86 Bisecting: 10693 revisions left to test after this (roughly 13 steps) [0db8640df59512dbd423c32077919f10cf35ebc6] Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next determine whether the revision contains the guilty commit revision c288ea679840de4dee2ce6da5d0f139e3774ad86 crashed and is reachable testing commit 0db8640df59512dbd423c32077919f10cf35ebc6 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 207ac2b0362d6eb7fd46e91e5b3656d04ce86f636ec2f3ad59df36461b3fe0fe all runs: OK false negative chance: 0.000 # git bisect bad 0db8640df59512dbd423c32077919f10cf35ebc6 Bisecting: 5403 revisions left to test after this (roughly 12 steps) [3c7c25038b6c7d66a6816028219914379be6a5cc] Merge tag 'block-5.17-2022-01-21' of git://git.kernel.dk/linux-block determine whether the revision contains the guilty commit revision c288ea679840de4dee2ce6da5d0f139e3774ad86 crashed and is reachable testing commit 3c7c25038b6c7d66a6816028219914379be6a5cc gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 73eef28df83348ca1b6f455af5c9bc7caf94c64424bb9fda0ea33db6f0e1480c all runs: OK false negative chance: 0.000 # git bisect bad 3c7c25038b6c7d66a6816028219914379be6a5cc Bisecting: 2630 revisions left to test after this (roughly 11 steps) [3ceff4ea07410763d5d4cccd60349bf7691e7e61] Merge tag 'sound-5.17-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound determine whether the revision contains the guilty commit revision c288ea679840de4dee2ce6da5d0f139e3774ad86 crashed and is reachable testing commit 3ceff4ea07410763d5d4cccd60349bf7691e7e61 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d0cbf415e577c8d3720ca3dc126ab03d096f2a6cdc74919dada5ab1bbbfabda8 run #0: infra problem: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS ErrorDetails:[0xc00067afa0 0xc00067b090 0xc00067b130] Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK false negative chance: 0.000 # git bisect bad 3ceff4ea07410763d5d4cccd60349bf7691e7e61 Bisecting: 1384 revisions left to test after this (roughly 11 steps) [6020c204be997e3f5129839ff9c801800fb4336e] Merge tag 'folio-5.17' of git://git.infradead.org/users/willy/pagecache determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 6020c204be997e3f5129839ff9c801800fb4336e gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d777165319f8607af1aed186c6811e230a3ebbdd73a8af1b2742709de3f831ce all runs: OK false negative chance: 0.000 # git bisect bad 6020c204be997e3f5129839ff9c801800fb4336e Bisecting: 645 revisions left to test after this (roughly 10 steps) [e3084ed48fd6b661fe434da0cb36d7d6706cf27f] Merge tag 'pinctrl-v5.17-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit e3084ed48fd6b661fe434da0cb36d7d6706cf27f gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a89b38f57fed73041985f19b9dd3dfc89983ad29f4d9b20304fb1cbd3ad14884 run #0: infra problem: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS ErrorDetails:[0xc0094a8af0 0xc0094a8be0 0xc0094a8c80] Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #1: infra problem: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS ErrorDetails:[0xc0019629b0 0xc001962b40 0xc001962be0] Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK false negative chance: 0.000 # git bisect bad e3084ed48fd6b661fe434da0cb36d7d6706cf27f Bisecting: 369 revisions left to test after this (roughly 9 steps) [3e3a138a46906641d95599040f2470a60740c399] Merge tag 'for-linus' of git://git.armlinux.org.uk/~rmk/linux-arm determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 3e3a138a46906641d95599040f2470a60740c399 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 0dae66c789f397fc88aef8fde1a9fe95b6e33725efc4dd2ab5261c3dc91f790d all runs: OK false negative chance: 0.000 # git bisect bad 3e3a138a46906641d95599040f2470a60740c399 Bisecting: 187 revisions left to test after this (roughly 8 steps) [d601e58c5f2901783428bc1181e83ff783592b6b] Merge tag 'for-5.17-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit d601e58c5f2901783428bc1181e83ff783592b6b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 4315e9ddbc98bdd8fcd7bbde4c3befdcc769e2d31aa2402f2578b9e6668226f6 run #0: infra problem: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS ErrorDetails:[0xc003ab1130 0xc003ab1220 0xc003ab12c0] Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #1: infra problem: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS ErrorDetails:[0xc003ae2be0 0xc003ae2cd0 0xc003ae2d70] Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #2: infra problem: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS ErrorDetails:[0xc003ab1400 0xc003ab14f0 0xc003ab1590] Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #3: infra problem: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS ErrorDetails:[0xc003ab1680 0xc003ab1770 0xc003ab1810] Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #4: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #5: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #6: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #7: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #8: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit run #9: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good d601e58c5f2901783428bc1181e83ff783592b6b Bisecting: 84 revisions left to test after this (roughly 7 steps) [3f67eaed57dae339603441cf0c0a74ec77a9fc03] Merge tag 'dlm-5.17' of git://git.kernel.org/pub/scm/linux/kernel/git/teigland/linux-dlm determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 3f67eaed57dae339603441cf0c0a74ec77a9fc03 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 4736395a983ffbe3e32e0f20f5e2385891d0e2036f8c271a0b5080b34fc87f26 all runs: OK false negative chance: 0.000 # git bisect bad 3f67eaed57dae339603441cf0c0a74ec77a9fc03 Bisecting: 54 revisions left to test after this (roughly 6 steps) [6eeaf88fd586f05aaf1d48cb3a139d2a5c6eb055] ext4: don't use the orphan list when migrating an inode determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 6eeaf88fd586f05aaf1d48cb3a139d2a5c6eb055 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: cb5851bbccc0a54dcc79472c84f6efc4fbbe09c87653727b416b8c7ffcea14b6 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good 6eeaf88fd586f05aaf1d48cb3a139d2a5c6eb055 Bisecting: 27 revisions left to test after this (roughly 5 steps) [21d9ac1a5376d949199398848006f6b14649f533] fs: dlm: use event based wait for pending remove determine whether the revision contains the guilty commit checking the merge base 3906fe9bb7f1a2c8667ae54e967dc8690824f4ea no existing result, test the revision testing commit 3906fe9bb7f1a2c8667ae54e967dc8690824f4ea gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 73039c5f1d931c5e4c7dd1c24055cae1961c3dad820d0d7a1cffbb088bbe4d93 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] testing commit 21d9ac1a5376d949199398848006f6b14649f533 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 73039c5f1d931c5e4c7dd1c24055cae1961c3dad820d0d7a1cffbb088bbe4d93 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good 21d9ac1a5376d949199398848006f6b14649f533 Bisecting: 13 revisions left to test after this (roughly 4 steps) [7e937bb3cbe1f6b9840a43f879aa6e3f1a5e6537] xfs: warn about inodes with project id of -1 determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 7e937bb3cbe1f6b9840a43f879aa6e3f1a5e6537 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: aa816bd9d431a168fdbcf6a05465e7672fba002cc721c871565f42a96f74db61 all runs: OK false negative chance: 0.000 # git bisect bad 7e937bb3cbe1f6b9840a43f879aa6e3f1a5e6537 Bisecting: 6 revisions left to test after this (roughly 3 steps) [132c460e49649685bf4b02ba43dea59062f797d9] xfs: Fix comments mentioning xfs_ialloc determine whether the revision contains the guilty commit revision 8bb7eca972ad531c9b149c0a51ab43a417385813 crashed and is reachable testing commit 132c460e49649685bf4b02ba43dea59062f797d9 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 93d3e88201c3997c1985e96483e24e88166a41acb86f872daeee0ca87ac27840 all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good 132c460e49649685bf4b02ba43dea59062f797d9 Bisecting: 3 revisions left to test after this (roughly 2 steps) [219aac5d469f7c0bc8ca299dcdeab70380076d24] xfs: sysfs: use default_groups in kobj_type determine whether the revision contains the guilty commit revision 3906fe9bb7f1a2c8667ae54e967dc8690824f4ea crashed and is reachable testing commit 219aac5d469f7c0bc8ca299dcdeab70380076d24 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6119cbacf6571df388b51c1ffc38521e1e05e0ebdba095d67f08cc7229fa408d all runs: crashed: KASAN: null-ptr-deref Write in xlog_cil_commit representative crash: KASAN: null-ptr-deref Write in xlog_cil_commit, types: [KASAN] # git bisect good 219aac5d469f7c0bc8ca299dcdeab70380076d24 Bisecting: 1 revision left to test after this (roughly 1 step) [f4901a182d33d05a3b7020e2af97c635f6c47959] xfs: Remove redundant assignment of mp determine whether the revision contains the guilty commit revision 132c460e49649685bf4b02ba43dea59062f797d9 crashed and is reachable testing commit f4901a182d33d05a3b7020e2af97c635f6c47959 gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2618ec7b4bb9b3b0a30fcfa1d5b76532134898a83c26dcb0128045b1aafb62f1 all runs: OK false negative chance: 0.000 # git bisect bad f4901a182d33d05a3b7020e2af97c635f6c47959 Bisecting: 0 revisions left to test after this (roughly 0 steps) [8dc9384b7d75012856b02ff44c37566a55fc2abf] xfs: reduce kvmalloc overhead for CIL shadow buffers determine whether the revision contains the guilty commit revision 219aac5d469f7c0bc8ca299dcdeab70380076d24 crashed and is reachable testing commit 8dc9384b7d75012856b02ff44c37566a55fc2abf gcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2618ec7b4bb9b3b0a30fcfa1d5b76532134898a83c26dcb0128045b1aafb62f1 all runs: OK false negative chance: 0.000 # git bisect bad 8dc9384b7d75012856b02ff44c37566a55fc2abf 8dc9384b7d75012856b02ff44c37566a55fc2abf is the first bad commit commit 8dc9384b7d75012856b02ff44c37566a55fc2abf Author: Dave Chinner Date: Tue Jan 4 17:22:18 2022 -0800 xfs: reduce kvmalloc overhead for CIL shadow buffers Oh, let me count the ways that the kvmalloc API sucks dog eggs. The problem is when we are logging lots of large objects, we hit kvmalloc really damn hard with costly order allocations, and behaviour utterly sucks: - 49.73% xlog_cil_commit - 31.62% kvmalloc_node - 29.96% __kmalloc_node - 29.38% kmalloc_large_node - 29.33% __alloc_pages - 24.33% __alloc_pages_slowpath.constprop.0 - 18.35% __alloc_pages_direct_compact - 17.39% try_to_compact_pages - compact_zone_order - 15.26% compact_zone 5.29% __pageblock_pfn_to_page 3.71% PageHuge - 1.44% isolate_migratepages_block 0.71% set_pfnblock_flags_mask 1.11% get_pfnblock_flags_mask - 0.81% get_page_from_freelist - 0.59% _raw_spin_lock_irqsave - do_raw_spin_lock __pv_queued_spin_lock_slowpath - 3.24% try_to_free_pages - 3.14% shrink_node - 2.94% shrink_slab.constprop.0 - 0.89% super_cache_count - 0.66% xfs_fs_nr_cached_objects - 0.65% xfs_reclaim_inodes_count 0.55% xfs_perag_get_tag 0.58% kfree_rcu_shrink_count - 2.09% get_page_from_freelist - 1.03% _raw_spin_lock_irqsave - do_raw_spin_lock __pv_queued_spin_lock_slowpath - 4.88% get_page_from_freelist - 3.66% _raw_spin_lock_irqsave - do_raw_spin_lock __pv_queued_spin_lock_slowpath - 1.63% __vmalloc_node - __vmalloc_node_range - 1.10% __alloc_pages_bulk - 0.93% __alloc_pages - 0.92% get_page_from_freelist - 0.89% rmqueue_bulk - 0.69% _raw_spin_lock - do_raw_spin_lock __pv_queued_spin_lock_slowpath 13.73% memcpy_erms - 2.22% kvfree On this workload, that's almost a dozen CPUs all trying to compact and reclaim memory inside kvmalloc_node at the same time. Yet it is regularly falling back to vmalloc despite all that compaction, page and shrinker reclaim that direct reclaim is doing. Copying all the metadata is taking far less CPU time than allocating the storage! Direct reclaim should be considered extremely harmful. This is a high frequency, high throughput, CPU usage and latency sensitive allocation. We've got memory there, and we're using kvmalloc to allow memory allocation to avoid doing lots of work to try to do contiguous allocations. Except it still does *lots of costly work* that is unnecessary. Worse: the only way to avoid the slowpath page allocation trying to do compaction on costly allocations is to turn off direct reclaim (i.e. remove __GFP_RECLAIM_DIRECT from the gfp flags). Unfortunately, the stupid kvmalloc API then says "oh, this isn't a GFP_KERNEL allocation context, so you only get kmalloc!". This cuts off the vmalloc fallback, and this leads to almost instant OOM problems which ends up in filesystems deadlocks, shutdowns and/or kernel crashes. I want some basic kvmalloc behaviour: - kmalloc for a contiguous range with fail fast semantics - no compaction direct reclaim if the allocation enters the slow path. - run normal vmalloc (i.e. GFP_KERNEL) if kmalloc fails The really, really stupid part about this is these kvmalloc() calls are run under memalloc_nofs task context, so all the allocations are always reduced to GFP_NOFS regardless of the fact that kvmalloc requires GFP_KERNEL to be passed in. IOWs, we're already telling kvmalloc to behave differently to the gfp flags we pass in, but it still won't allow vmalloc to be run with anything other than GFP_KERNEL. So, this patch open codes the kvmalloc() in the commit path to have the above described behaviour. The result is we more than halve the CPU time spend doing kvmalloc() in this path and transaction commits with 64kB objects in them more than doubles. i.e. we get ~5x reduction in CPU usage per costly-sized kvmalloc() invocation and the profile looks like this: - 37.60% xlog_cil_commit 16.01% memcpy_erms - 8.45% __kmalloc - 8.04% kmalloc_order_trace - 8.03% kmalloc_order - 7.93% alloc_pages - 7.90% __alloc_pages - 4.05% __alloc_pages_slowpath.constprop.0 - 2.18% get_page_from_freelist - 1.77% wake_all_kswapds .... - __wake_up_common_lock - 0.94% _raw_spin_lock_irqsave - 3.72% get_page_from_freelist - 2.43% _raw_spin_lock_irqsave - 5.72% vmalloc - 5.72% __vmalloc_node_range - 4.81% __get_vm_area_node.constprop.0 - 3.26% alloc_vmap_area - 2.52% _raw_spin_lock - 1.46% _raw_spin_lock 0.56% __alloc_pages_bulk - 4.66% kvfree - 3.25% vfree - __vfree - 3.23% __vunmap - 1.95% remove_vm_area - 1.06% free_vmap_area_noflush - 0.82% _raw_spin_lock - 0.68% _raw_spin_lock - 0.92% _raw_spin_lock - 1.40% kfree - 1.36% __free_pages - 1.35% __free_pages_ok - 1.02% _raw_spin_lock_irqsave It's worth noting that over 50% of the CPU time spent allocating these shadow buffers is now spent on spinlocks. So the shadow buffer allocation overhead is greatly reduced by getting rid of direct reclaim from kmalloc, and could probably be made even less costly if vmalloc() didn't use global spinlocks to protect it's structures. Signed-off-by: Dave Chinner Reviewed-by: Allison Henderson Reviewed-by: Darrick J. Wong Signed-off-by: Darrick J. Wong fs/xfs/xfs_log_cil.c | 46 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 11 deletions(-) accumulated error probability: 0.00 culprit signature: 2618ec7b4bb9b3b0a30fcfa1d5b76532134898a83c26dcb0128045b1aafb62f1 parent signature: 6119cbacf6571df388b51c1ffc38521e1e05e0ebdba095d67f08cc7229fa408d revisions tested: 28, total time: 4h55m5.277142678s (build: 1h11m40.463309887s, test: 2h50m51.811074376s) first good commit: 8dc9384b7d75012856b02ff44c37566a55fc2abf xfs: reduce kvmalloc overhead for CIL shadow buffers recipients (to): ["allison.henderson@oracle.com" "dchinner@redhat.com" "djwong@kernel.org"] recipients (cc): []