ci starts bisection 2024-05-07 12:56:33.211509785 +0000 UTC m=+2943.644227972 bisecting cause commit starting from ee5b455b0adae9ecafb38b174c648c48f2a3c1a5 building syzkaller on c035c6de7b75d2ca88982bf059fd4e27cbad7a26 ensuring issue is reproducible on original commit ee5b455b0adae9ecafb38b174c648c48f2a3c1a5 testing commit ee5b455b0adae9ecafb38b174c648c48f2a3c1a5 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3c4199cf9fa080ad4197d6fdefee27e19cde4256a83c0aa09ede5e432872e5ed run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: rcu detected stall in llc_rcv run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in llc_rcv run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: rcu detected stall in llc_rcv run #6: crashed: INFO: rcu detected stall in sys_bpf run #7: crashed: INFO: rcu detected stall in llc_rcv run #8: crashed: INFO: rcu detected stall in team_handle_frame run #9: crashed: INFO: rcu detected stall in sys_bpf run #10: crashed: INFO: rcu detected stall in handle_softirqs run #11: crashed: INFO: rcu detected stall in corrupted run #12: crashed: INFO: rcu detected stall in corrupted run #13: crashed: INFO: rcu detected stall in corrupted run #14: crashed: INFO: rcu detected stall in corrupted run #15: crashed: INFO: rcu detected stall in corrupted run #16: crashed: INFO: rcu detected stall in addrconf_dad_work run #17: crashed: INFO: rcu detected stall in corrupted run #18: crashed: INFO: rcu detected stall in llc_rcv run #19: crashed: INFO: rcu detected stall in llc_rcv representative crash: INFO: rcu detected stall in corrupted, types: [HANG] check whether we can drop unnecessary instrumentation disabling configs for [ATOMIC_SLEEP LEAK UBSAN BUG KASAN LOCKDEP], they are not needed testing commit ee5b455b0adae9ecafb38b174c648c48f2a3c1a5 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 00b00bc9a52a81784dbffbe5a0c22e7e117b503492141a42e1a0e2a02f3c3f6f run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: rcu detected stall in sys_bpf run #2: crashed: INFO: rcu detected stall in sys_bpf run #3: crashed: INFO: rcu detected stall in corrupted run #4: crashed: INFO: rcu detected stall in sys_bpf run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: rcu detected stall in corrupted run #7: crashed: INFO: rcu detected stall in sys_bpf run #8: crashed: INFO: rcu detected stall in sys_bpf run #9: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] the bug reproduces without the instrumentation disabling configs for [LOCKDEP ATOMIC_SLEEP LEAK UBSAN BUG KASAN], they are not needed kconfig minimization: base=3979 full=8006 leaves diff=2010 split chunks (needed=false): <2010> split chunk #0 of len 2010 into 5 parts testing without sub-chunk 1/5 disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP LEAK UBSAN BUG], they are not needed testing commit ee5b455b0adae9ecafb38b174c648c48f2a3c1a5 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 99659e457af24e7fe7bf396a096d08810268f0754b398f8757950c2d97055369 run #0: crashed: INFO: rcu detected stall in gc_worker run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in corrupted run #4: crashed: INFO: rcu detected stall in llc_rcv run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: rcu detected stall in corrupted run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: INFO: rcu detected stall in handle_softirqs run #9: crashed: INFO: rcu detected stall in handle_softirqs representative crash: INFO: rcu detected stall in gc_worker, types: [HANG] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit ee5b455b0adae9ecafb38b174c648c48f2a3c1a5 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 7edaa659c51c892a32d9a33459a27f397285ce1eb8e3b3e131c33c51c74cddbd all runs: OK false negative chance: 0.000 testing without sub-chunk 3/5 disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit ee5b455b0adae9ecafb38b174c648c48f2a3c1a5 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: fb32ed9a80d4ba799875bb03f79ef065ae669abf8a966374e42ad13d2f61e286 run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in corrupted run #4: crashed: INFO: rcu detected stall in handle_softirqs run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: rcu detected stall in sys_bpf run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: INFO: rcu detected stall in corrupted run #9: crashed: INFO: rcu detected stall in handle_softirqs representative crash: INFO: rcu detected stall in corrupted, types: [HANG] the chunk can be dropped testing without sub-chunk 4/5 disabling configs for [ATOMIC_SLEEP LEAK UBSAN BUG KASAN LOCKDEP], they are not needed testing commit ee5b455b0adae9ecafb38b174c648c48f2a3c1a5 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: c64ead2eb911357d3c997597dc7378b4912e0b07e32a21061c9f22f29a8cf946 all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [ATOMIC_SLEEP LEAK UBSAN BUG KASAN LOCKDEP], they are not needed testing commit ee5b455b0adae9ecafb38b174c648c48f2a3c1a5 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 0ccbb5077ed19a474e3c430875915f39486218724d9b96348b0bda509368ac9d all runs: OK false negative chance: 0.000 minimized to 804 configs; suspects: [6LOWPAN ARCH_ENABLE_MEMORY_HOTREMOVE ASUS_WMI ATM BCMA BLK_DEV_ZONED BPF_SYSCALL CARDBUS CFG80211 CFG80211_WEXT CMA COMMON_CLK CONTIG_ALLOC CRYPTO_842 CRYPTO_LZ4 CRYPTO_LZ4HC CRYPTO_ZSTD DAX DLM DRM DRM_BRIDGE DRM_CIRRUS_QEMU DRM_DEBUG_MM DRM_DISPLAY_DP_HELPER DRM_DISPLAY_HDCP_HELPER DRM_DISPLAY_HDMI_HELPER DRM_DISPLAY_HELPER DRM_DP_AUX_BUS DRM_FBDEV_EMULATION DRM_GEM_SHMEM_HELPER DRM_I915 DRM_I915_CAPTURE_ERROR DRM_I915_COMPRESS_ERROR DRM_I915_USERPTR DRM_KMS_HELPER DRM_MIPI_DSI DRM_PANEL DRM_PANEL_BRIDGE DRM_PANEL_EDP DRM_PANEL_ORIENTATION_QUIRKS DRM_SIMPLEDRM DRM_TTM DRM_TTM_HELPER DRM_UDL DRM_VGEM DRM_VIRTIO_GPU DRM_VIRTIO_GPU_KMS DRM_VKMS DRM_VMWGFX DRM_VRAM_HELPER DUMMY DVB_AF9013 DVB_AF9033 DVB_AS102 DVB_AS102_FE DVB_B2C2_FLEXCOP DVB_B2C2_FLEXCOP_USB DVB_CORE DVB_DIB3000MB DVB_DIB3000MC DVB_EC100 DVB_GP8PSK_FE DVB_RTL2830 DVB_RTL2832 DVB_RTL2832_SDR DVB_TEST_DRIVERS DVB_TTUSB_BUDGET DVB_TTUSB_DEC DVB_USB DVB_USB_A800 DVB_USB_AF9005 DVB_USB_AF9005_REMOTE DVB_USB_AF9015 DVB_USB_AF9035 DVB_USB_ANYSEE DVB_USB_AU6610 DVB_USB_AZ6007 DVB_USB_AZ6027 DVB_USB_CE6230 DVB_USB_CINERGY_T2 DVB_USB_CXUSB DVB_USB_DIB0700 DVB_USB_DIB3000MC DVB_USB_DIBUSB_MB DVB_USB_DIBUSB_MC DVB_USB_DIGITV DVB_USB_DTT200U DVB_USB_DTV5100 DVB_USB_DVBSKY DVB_USB_DW2102 DVB_USB_EC168 DVB_USB_GL861 DVB_USB_GP8PSK DVB_USB_LME2510 DVB_USB_M920X DVB_USB_MXL111SF DVB_USB_NOVA_T_USB2 DVB_USB_OPERA1 DVB_USB_PCTV452E DVB_USB_RTL28XXU DVB_USB_TECHNISAT_USB2 DVB_USB_TTUSB2 DVB_USB_UMT_010 DVB_USB_V2 DVB_USB_VP702X DVB_USB_VP7045 DVB_USB_ZD1301 DVB_VIDTV DVB_ZL10353 ECRYPT_FS ECRYPT_FS_MESSAGING EDAC EEPROM_93CX6 EFS_FS ENCRYPTED_KEYS EQUALIZER EROFS_FS EROFS_FS_POSIX_ACL EROFS_FS_SECURITY EROFS_FS_XATTR EROFS_FS_ZIP EVM EVM_ADD_XATTRS EVM_ATTR_FSUUID EXFAT_FS EXPORTFS_BLOCK_OPS EXT3_FS EXT3_FS_POSIX_ACL EXT3_FS_SECURITY EXTCON EXTCON_INTEL_CHT_WC F2FS_CHECK_FS F2FS_FAULT_INJECTION F2FS_FS F2FS_FS_COMPRESSION F2FS_FS_LZ4 F2FS_FS_LZ4HC F2FS_FS_LZO F2FS_FS_LZORLE F2FS_FS_POSIX_ACL F2FS_FS_SECURITY F2FS_FS_XATTR F2FS_FS_ZSTD F2FS_STAT_FS FANOTIFY FANOTIFY_ACCESS_PERMISSIONS FB FB_CFB_COPYAREA FB_CFB_FILLRECT FB_CFB_IMAGEBLIT FB_CORE FB_DEFERRED_IO FB_IOMEM_FOPS FB_IOMEM_HELPERS FB_NOTIFY FB_SYSMEM_FOPS FB_SYSMEM_HELPERS FB_SYSMEM_HELPERS_DEFERRED FB_SYS_COPYAREA FB_SYS_FILLRECT FB_SYS_IMAGEBLIT FB_TILEBLITTING FB_VESA FB_VGA16 FB_VIRTUAL FDDI FIREWIRE FIREWIRE_NET FIREWIRE_OHCI FIREWIRE_SBP2 FONT_8x16 FONT_8x8 FONT_SUPPORT FRAMEBUFFER_CONSOLE FRAMEBUFFER_CONSOLE_DETECT_PRIMARY FRAMEBUFFER_CONSOLE_ROTATION FS_DAX FS_DAX_PMD FS_ENCRYPTION FS_ENCRYPTION_ALGS FS_STACK FS_VERITY FS_VERITY_BUILTIN_SIGNATURES FTL FUSE_DAX FUSE_FS FW_LOADER_COMPRESS FW_LOADER_PAGED_BUF FW_LOADER_SYSFS FW_LOADER_USER_HELPER FW_LOADER_USER_HELPER_FALLBACK GACT_PROB GARP GENERIC_PHY GET_FREE_REGION GFS2_FS GFS2_FS_LOCKING_DLM GOOGLE_COREBOOT_TABLE GOOGLE_FIRMWARE GOOGLE_MEMCONSOLE GOOGLE_MEMCONSOLE_COREBOOT GOOGLE_VPD GPIOLIB GPIOLIB_IRQCHIP GPIO_ACPI GPIO_DLN2 GPIO_VIPERBOARD GREENASIA_FF GREYBUS GREYBUS_BRIDGED_PHY GREYBUS_ES2 GREYBUS_HID GREYBUS_USB GTP GUEST_PERF_EVENTS GVE HAVE_ARCH_NODE_DEV_GROUP HAVE_ARCH_USERFAULTFD_MINOR HAVE_ARCH_USERFAULTFD_WP HAVE_BOOTMEM_INFO_NODE HAVE_CLK_PREPARE HAVE_KVM_CPU_RELAX_INTERCEPT HAVE_KVM_DIRTY_RING HAVE_KVM_DIRTY_RING_ACQ_REL HAVE_KVM_DIRTY_RING_TSO HAVE_KVM_IRQCHIP HAVE_KVM_IRQ_BYPASS HAVE_KVM_IRQ_ROUTING HAVE_KVM_MSI HAVE_KVM_NO_POLL HAVE_KVM_PFNCACHE HAVE_KVM_PM_NOTIFIER HAVE_KVM_READONLY_MEM HAVE_SCHED_AVG_IRQ HDLC HDLC_CISCO HDLC_FR HDLC_PPP HDLC_RAW HDLC_RAW_ETH HDLC_X25 HDMI HFSPLUS_FS HFS_FS HID_ACCUTOUCH HID_ACRUX HID_ACRUX_FF HID_ALPS HID_APPLEIR HID_ASUS HID_AUREAL HID_BATTERY_STRENGTH HID_BETOP_FF HID_CMEDIA HID_CORSAIR HID_CP2112 HID_ELECOM HID_ELO HID_EMS_FF HID_GEMBIRD HID_GFRM HID_GREENASIA HID_GT683R HID_HOLTEK HID_ICADE HID_KEYTOUCH HID_KYE HID_LCPOWER HID_LED HID_LENOVO HID_LOGITECH_DJ HID_LOGITECH_HIDPP HID_MAGICMOUSE HID_MAYFLASH HID_MULTITOUCH HID_NTI HID_ORTEK HID_PENMOUNT HID_PICOLCD HID_PICOLCD_BACKLIGHT HID_PICOLCD_CIR HID_PICOLCD_FB HID_PICOLCD_LCD HID_PICOLCD_LEDS HID_PLANTRONICS HID_PRIMAX HID_PRODIKEYS HID_RETRODE HID_RMI HID_ROCCAT HID_SAITEK HID_SENSOR_ACCEL_3D HID_SENSOR_ALS HID_SENSOR_CUSTOM_SENSOR HID_SENSOR_DEVICE_ROTATION HID_SENSOR_GYRO_3D HID_SENSOR_HUB HID_SENSOR_HUMIDITY HID_SENSOR_IIO_COMMON HID_SENSOR_IIO_TRIGGER HID_SENSOR_INCLINOMETER_3D HID_SENSOR_MAGNETOMETER_3D HID_SENSOR_PRESS HID_SENSOR_PROX HID_SENSOR_TEMP HID_SPEEDLINK HID_STEELSERIES HID_THINGM HID_TIVO HID_TWINHAN HID_UCLOGIC HID_UDRAW_PS3 HID_WACOM HID_WALTOP HID_WIIMOTE HID_XINMO HID_ZEROPLUS HID_ZYDACRON HMM_MIRROR HOLTEK_FF HOTPLUG_PCI_PCIE HPET_MMAP HPET_MMAP_DEFAULT HPFS_FS I2C_ALGOBIT I2C_CHARDEV I2C_DESIGNWARE_CORE I2C_DESIGNWARE_PLATFORM I2C_DIOLAN_U2C I2C_DLN2 I2C_MUX I2C_MUX_REG I2C_ROBOTFUZZ_OSIF I2C_SI4713 I2C_SLAVE I2C_SLAVE_EEPROM I2C_TINY_USB I2C_VIPERBOARD IEEE802154 IEEE802154_6LOWPAN IEEE802154_ATUSB IEEE802154_DRIVERS IEEE802154_HWSIM IEEE802154_NL802154_EXPERIMENTAL IEEE802154_SOCKET IFB IIO IIO_BUFFER IIO_KFIFO_BUF IIO_TRIGGER IIO_TRIGGERED_BUFFER IKCONFIG IKCONFIG_PROC IMA IMA_APPRAISE IMA_APPRAISE_MODSIG IMA_DEFAULT_HASH_SHA256 IMA_LSM_RULES IMA_MEASURE_ASYMMETRIC_KEYS IMA_NG_TEMPLATE IMA_QUEUE_EARLY_BOOT_KEYS IMA_READ_POLICY IMA_WRITE_POLICY INET6_ESPINTCP INET6_ESP_OFFLOAD INET6_IPCOMP INET6_TUNNEL INET6_XFRM_TUNNEL INET_AH INET_DCCP_DIAG INET_DIAG INET_DIAG_DESTROY INET_ESP INET_ESPINTCP INET_ESP_OFFLOAD INET_IPCOMP INET_MPTCP_DIAG INET_RAW_DIAG INET_SCTP_DIAG INET_TCP_DIAG INET_UDP_DIAG INET_XFRM_TUNNEL INFINIBAND INFINIBAND_ADDR_TRANS INFINIBAND_ADDR_TRANS_CONFIGFS INFINIBAND_IPOIB INFINIBAND_IPOIB_CM INFINIBAND_IPOIB_DEBUG INFINIBAND_ISER INFINIBAND_ON_DEMAND_PAGING INFINIBAND_RTRS INFINIBAND_SRP INFINIBAND_USER_ACCESS INFINIBAND_USER_MAD INFINIBAND_USER_MEM INPUT_ATI_REMOTE2 INPUT_CM109 INPUT_IMS_PCU INPUT_JOYDEV INPUT_KEYSPAN_REMOTE INPUT_LEDS INPUT_MOUSEDEV INPUT_MOUSEDEV_PSAUX INPUT_POWERMATE INPUT_UINPUT INPUT_YEALINK INTEGRITY INTEGRITY_ASYMMETRIC_KEYS INTEGRITY_AUDIT INTEGRITY_SIGNATURE INTEGRITY_TRUSTED_KEYRING INTEL_IDMA64 INTEL_IOATDMA INTEL_IOMMU_DEFAULT_ON INTEL_IOMMU_SVM INTEL_ISH_HID INTEL_SOC_PMIC_CHTWC INTERVAL_TREE INTERVAL_TREE_SPAN_ITER IOMMUFD IOMMUFD_DRIVER IOMMUFD_TEST IOMMU_IOPF IOMMU_MM_DATA IOMMU_SVA IP6_NF_MATCH_AH IP6_NF_MATCH_EUI64 IP6_NF_MATCH_FRAG IP6_NF_MATCH_HL IP6_NF_MATCH_MH IP6_NF_MATCH_OPTS IP6_NF_MATCH_RPFILTER IP6_NF_MATCH_RT IP6_NF_MATCH_SRH IP6_NF_NAT IP6_NF_RAW IP6_NF_SECURITY IP6_NF_TARGET_HL IP6_NF_TARGET_MASQUERADE IP6_NF_TARGET_NPT IP6_NF_TARGET_SYNPROXY IPV6_FOU IPV6_FOU_TUNNEL IPV6_GRE IPV6_ILA IPV6_MIP6 IPV6_MROUTE IPV6_MROUTE_MULTIPLE_TABLES IPV6_MULTIPLE_TABLES IPV6_OPTIMISTIC_DAD IPV6_PIMSM_V2 IPV6_ROUTER_PREF IPV6_ROUTE_INFO IPV6_RPL_LWTUNNEL IPV6_SEG6_BPF IPV6_SEG6_HMAC IPV6_SEG6_LWTUNNEL IPV6_SIT_6RD IPV6_SUBTREES IPV6_TUNNEL IPV6_VTI IPVLAN IPVLAN_L3S IPVTAP IP_DCCP IP_DCCP_CCID3 IP_DCCP_TFRC_LIB IP_FIB_TRIE_STATS IP_MROUTE_MULTIPLE_TABLES IP_NF_ARPFILTER IP_NF_ARPTABLES IP_NF_ARP_MANGLE IP_NF_MATCH_AH IP_NF_MATCH_ECN IP_NF_MATCH_RPFILTER IP_NF_MATCH_TTL IP_NF_RAW IP_NF_SECURITY IP_NF_TARGET_ECN IP_NF_TARGET_NETMAP IP_NF_TARGET_REDIRECT IP_NF_TARGET_SYNPROXY IP_NF_TARGET_TTL IP_ROUTE_CLASSID IP_SCTP IP_SET IP_SET_BITMAP_IP IP_SET_BITMAP_IPMAC IP_SET_BITMAP_PORT IP_SET_HASH_IP IP_SET_HASH_IPMAC IP_SET_HASH_IPMARK IP_SET_HASH_IPPORT IP_SET_HASH_IPPORTIP IP_SET_HASH_IPPORTNET IP_SET_HASH_MAC IP_SET_HASH_NET IP_SET_HASH_NETIFACE IP_SET_HASH_NETNET IP_SET_HASH_NETPORT IP_SET_HASH_NETPORTNET IP_SET_LIST_SET IP_VS IP_VS_DH IP_VS_FO IP_VS_FTP IP_VS_IPV6 IP_VS_LBLC IP_VS_LBLCR IP_VS_LC IP_VS_MH IP_VS_PROTO_TCP IRQ_REMAP IRQ_TIME_ACCOUNTING KVM KVM_INTEL LAPB LCD_CLASS_DEVICE LIBNVDIMM MAC802154 MEDIA_ANALOG_TV_SUPPORT MEDIA_CAMERA_SUPPORT MEDIA_CEC_SUPPORT MEDIA_CONTROLLER MEDIA_DIGITAL_TV_SUPPORT MEDIA_RADIO_SUPPORT MEDIA_SDR_SUPPORT MEDIA_SUPPORT MEDIA_TEST_SUPPORT MEDIA_USB_SUPPORT MEMORY_HOTPLUG MEMORY_HOTREMOVE MFD_DLN2 MFD_VIPERBOARD MPTCP MTD NETFILTER_ADVANCED NET_ACT_GACT NET_ACT_MIRRED NET_IPGRE_DEMUX NFT_COMPAT NFT_COMPAT_ARP NFT_FWD_NETDEV NF_TABLES NF_TABLES_ARP NF_TABLES_NETDEV PARPORT PCCARD PCMCIA PHONET RADIO_ADAPTERS RADIO_SI470X RADIO_SI4713 RAS RC_CORE RFKILL SND SOUND SPI SSB STAGING TAP TARGET_CORE TRANSPARENT_HUGEPAGE TUN USB_AMD5536UDC USB_ATM USB_CONFIGFS USB_CONFIGFS_ECM USB_CONFIGFS_ECM_SUBSET USB_CONFIGFS_EEM USB_CONFIGFS_F_FS USB_CONFIGFS_F_HID USB_CONFIGFS_F_LB_SS USB_CONFIGFS_F_MIDI USB_CONFIGFS_F_PRINTER USB_CONFIGFS_F_TCM USB_CONFIGFS_F_UAC1 USB_CONFIGFS_F_UAC1_LEGACY USB_CONFIGFS_F_UAC2 USB_CONFIGFS_F_UVC USB_CONFIGFS_MASS_STORAGE USB_CONFIGFS_NCM USB_CONFIGFS_OBEX USB_CONFIGFS_PHONET USB_CONFIGFS_RNDIS USB_CONFIGFS_SERIAL USB_CXACRU USB_CYPRESS_CY7C63 USB_CYTHERM USB_DSBR USB_DUMMY_HCD USB_DWC2 USB_DWC2_HOST USB_DWC2_PCI USB_DWC3 USB_DWC3_GADGET USB_DWC3_OF_SIMPLE USB_DWC3_PCI USB_DWC3_ULPI USB_DYNAMIC_MINORS USB_EG20T USB_EHCI_HCD_PLATFORM USB_EHCI_ROOT_HUB_TT USB_EHSET_TEST_FIXTURE USB_EMI26 USB_EMI62 USB_EPSON2888 USB_EZUSB_FX2 USB_FEW_INIT_RETRIES USB_F_ACM USB_F_ECM USB_F_EEM USB_F_FS USB_F_HID USB_F_MASS_STORAGE USB_F_MIDI USB_F_NCM USB_F_OBEX USB_F_PHONET USB_F_PRINTER USB_F_RNDIS USB_F_SERIAL USB_F_SS_LB USB_F_SUBSET USB_F_TCM USB_F_UAC1 USB_F_UAC1_LEGACY USB_F_UAC2 USB_F_UVC USB_GADGET USB_GADGETFS USB_GADGET_DEBUG_FILES USB_GADGET_DEBUG_FS USB_GL860 USB_GOKU USB_GPIO_VBUS USB_GR_UDC USB_GSPCA USB_GSPCA_BENQ USB_GSPCA_CONEX USB_GSPCA_CPIA1 USB_GSPCA_DTCS033 USB_GSPCA_ETOMS USB_GSPCA_FINEPIX USB_GSPCA_JEILINJ USB_GSPCA_JL2005BCD USB_GSPCA_KINECT USB_GSPCA_KONICA USB_GSPCA_MARS USB_GSPCA_MR97310A USB_GSPCA_NW80X USB_GSPCA_OV519 USB_GSPCA_OV534 USB_GSPCA_OV534_9 USB_GSPCA_PAC207 USB_GSPCA_PAC7302 USB_GSPCA_PAC7311 USB_GSPCA_SE401 USB_GSPCA_SN9C2028 USB_GSPCA_SN9C20X USB_GSPCA_SONIXB USB_GSPCA_SONIXJ USB_GSPCA_SPCA1528 USB_GSPCA_SPCA500 USB_GSPCA_SPCA501 USB_GSPCA_SPCA505 USB_GSPCA_SPCA506 USB_GSPCA_SPCA508 USB_GSPCA_SPCA561 USB_GSPCA_SQ905 USB_GSPCA_SQ905C USB_GSPCA_SQ930X USB_GSPCA_STK014 USB_GSPCA_STK1135 USB_GSPCA_STV0680 USB_GSPCA_SUNPLUS USB_GSPCA_T613 USB_GSPCA_TOPRO USB_GSPCA_TOUPTEK USB_GSPCA_TV8532 USB_GSPCA_VC032X USB_GSPCA_VICAM USB_GSPCA_XIRLINK_CIT USB_GSPCA_ZC3XX USB_HACKRF USB_HCD_BCMA USB_HCD_SSB USB_HSIC_USB3503 USB_HSIC_USB4604 USB_HSO USB_HUB_USB251XB USB_IDMOUSE USB_IOWARRIOR USB_IPHETH USB_ISIGHTFW USB_ISP116X_HCD USB_ISP1301 USB_ISP1760 USB_ISP1760_DUAL_ROLE USB_ISP1760_HCD USB_ISP1761_UDC USB_KAWETH USB_KC2190 USB_KEENE USB_LAN78XX USB_LCD USB_LD USB_LEDS_TRIGGER_USBPORT USB_LED_TRIG USB_LEGOTOWER USB_LIBCOMPOSITE USB_LINK_LAYER_TEST USB_M5602 USB_MA901 USB_MAX3421_HCD USB_MDC800 USB_MICROTEK USB_MR800 USB_MSI2500 USB_MUSB_DUAL_ROLE USB_MUSB_HDRC USB_MV_U3D USB_MV_UDC USB_NET2272 USB_NET2272_DMA USB_NET2280 USB_NET_AX88179_178A USB_NET_AX8817X USB_NET_CDCETHER USB_NET_CDC_EEM USB_NET_CDC_MBIM USB_NET_CDC_NCM USB_NET_CDC_SUBSET USB_NET_CDC_SUBSET_ENABLE USB_NET_CH9200 USB_NET_CX82310_ETH USB_NET_DM9601 USB_NET_GL620A USB_NET_HUAWEI_CDC_NCM USB_NET_INT51X1 USB_NET_KALMIA USB_NET_MCS7830 USB_NET_NET1080 USB_NET_PLUSB USB_NET_QMI_WWAN USB_NET_RNDIS_HOST USB_NET_SMSC75XX USB_NET_SMSC95XX USB_NET_SR9700 USB_NET_SR9800 USB_NET_ZAURUS USB_OHCI_HCD_PLATFORM USB_OTG USB_OTG_FSM USB_OXU210HP_HCD USB_PEGASUS USB_PULSE8_CEC USB_PWC USB_PWC_INPUT_EVDEV USB_PXA27X USB_R8A66597 USB_R8A66597_HCD USB_RAINSHADOW_CEC USB_RAREMONO USB_RAW_GADGET USB_RTL8150 USB_RTL8152 USB_RTL8153_ECM USB_S2255 USB_SERIAL USB_SERIAL_AIRCABLE USB_SERIAL_ARK3116 USB_SERIAL_BELKIN USB_SERIAL_CH341 USB_SERIAL_CONSOLE USB_SERIAL_CP210X USB_SERIAL_CYBERJACK USB_SERIAL_CYPRESS_M8 USB_SERIAL_DEBUG USB_SERIAL_DIGI_ACCELEPORT USB_SERIAL_EDGEPORT USB_SERIAL_EDGEPORT_TI USB_SERIAL_EMPEG USB_SERIAL_F81232 USB_SERIAL_F8153X USB_SERIAL_FTDI_SIO USB_SERIAL_GARMIN USB_SERIAL_GENERIC USB_SERIAL_IPAQ USB_SERIAL_IPW USB_SERIAL_IR USB_SERIAL_IUU USB_SERIAL_KEYSPAN USB_SERIAL_KEYSPAN_PDA USB_SERIAL_KLSI USB_SERIAL_KOBIL_SCT USB_SERIAL_MCT_U232 USB_SERIAL_METRO USB_SERIAL_MOS7715_PARPORT USB_SERIAL_MOS7720 USB_SERIAL_MOS7840 USB_SERIAL_MXUPORT USB_SERIAL_NAVMAN USB_SERIAL_OMNINET USB_SERIAL_OPTICON USB_SERIAL_OPTION USB_SERIAL_OTI6858 USB_SERIAL_PL2303 USB_SERIAL_QCAUX USB_SERIAL_QT2 USB_SERIAL_QUALCOMM USB_SERIAL_SAFE USB_SERIAL_SIERRAWIRELESS USB_SERIAL_SIMPLE USB_SERIAL_SPCP8X5 USB_SERIAL_SSU100 USB_SERIAL_SYMBOL USB_SERIAL_TI USB_SERIAL_UPD78F0730 USB_SERIAL_VISOR USB_SERIAL_WHITEHEAT USB_SERIAL_WISHBONE USB_SERIAL_WWAN USB_SERIAL_XR USB_SERIAL_XSENS_MT USB_SEVSEG USB_SI470X USB_SI4713 USB_SIERRA_NET USB_SISUSBVGA USB_SL811_CS USB_SL811_HCD USB_SL811_HCD_ISO USB_SNP_CORE USB_SPEEDTOUCH USB_STORAGE_ALAUDA USB_STORAGE_CYPRESS_ATACB USB_STORAGE_DATAFAB USB_STORAGE_ENE_UB6250 USB_STORAGE_FREECOM USB_STORAGE_ISD200 USB_STORAGE_JUMPSHOT USB_STORAGE_KARMA USB_STORAGE_ONETOUCH USB_STORAGE_SDDR09 USB_STORAGE_SDDR55 USB_STORAGE_USBAT USB_STV06XX USB_TEST USB_TMC USB_TRANCEVIBRATOR USB_UAS USB_UEAGLEATM USB_ULPI_BUS USB_USBNET USB_USS720 USB_U_AUDIO USB_U_ETHER USB_U_SERIAL USB_VIDEO_CLASS USB_VIDEO_CLASS_INPUT_EVDEV USB_VL600 USB_WDM USB_XHCI_DBGCAP USB_XHCI_PLATFORM USB_XUSBATM USB_YUREX USERFAULTFD USERIO USERMODE_DRIVER USER_RETURN_NOTIFIER UVC_COMMON U_SERIAL_CONSOLE V4L2_MEM2MEM_DEV V4L_TEST_DRIVERS VALIDATE_FS_PARSER VDPA VDPA_SIM VDPA_SIM_BLOCK VDPA_SIM_NET VDPA_USER VETH VFIO VFIO_DEVICE_CDEV VFIO_PCI VFIO_PCI_CORE VFIO_PCI_INTX VFIO_PCI_MMAP VFIO_VIRQFD VGASTATE VHOST VHOST_CROSS_ENDIAN_LEGACY VHOST_IOTLB VHOST_NET VHOST_RING VHOST_TASK VHOST_VDPA VHOST_VSOCK VIDEO VIDEOBUF2_CORE VIDEOBUF2_DMA_CONTIG VIDEOBUF2_DMA_SG VIDEOBUF2_MEMOPS VIDEOBUF2_V4L2 VIDEOBUF2_VMALLOC VIDEOMODE_HELPERS VIDEO_AU0828 VIDEO_AU0828_RC VIDEO_AU0828_V4L2 VIDEO_CS53L32A VIDEO_CX231XX VIDEO_CX231XX_ALSA VIDEO_CX231XX_DVB VIDEO_CX231XX_RC VIDEO_CX2341X VIDEO_CX25840 VIDEO_DEV VIDEO_EM28XX VIDEO_EM28XX_ALSA VIDEO_EM28XX_DVB VIDEO_EM28XX_RC VIDEO_EM28XX_V4L2 VIDEO_GO7007 VIDEO_GO7007_LOADER VIDEO_GO7007_USB VIDEO_GO7007_USB_S2250_BOARD VIDEO_HDPVR VIDEO_MSP3400 VIDEO_PVRUSB2 VIDEO_PVRUSB2_DVB VIDEO_PVRUSB2_SYSFS VIDEO_SAA711X VIDEO_STK1160 VIDEO_TUNER VIDEO_TVEEPROM VIDEO_USBTV VIDEO_V4L2_I2C VIDEO_V4L2_SUBDEV_API VIDEO_V4L2_TPG VIDEO_VICODEC VIDEO_VIM2M VIDEO_VIMC VIDEO_VIVID VIDEO_VIVID_CEC VIDEO_WM8775 VIPERBOARD_ADC VIRTIO_BALLOON VIRTIO_DMA_SHARED_BUFFER VIRTIO_FS VIRTIO_MEM VIRTIO_MMIO VIRTIO_MMIO_CMDLINE_DEVICES VIRTIO_PMEM VIRTIO_VDPA VIRTIO_VSOCKETS VIRTIO_VSOCKETS_COMMON VIRT_WIFI VLAN_8021Q VLAN_8021Q_GVRP VLAN_8021Q_MVRP VMAP_PFN VMWARE_VMCI VMXNET3 VP_VDPA VSOCKETS VSOCKETS_DIAG VSOCKETS_LOOPBACK VSOCKMON VT_HW_CONSOLE_BINDING VXFS_FS WAN WANT_DEV_COREDUMP WEXT_CORE WEXT_PRIV WEXT_PROC WIREGUARD WIRELESS WIRELESS_EXT WLAN WLAN_VENDOR_ADMTEK WLAN_VENDOR_PURELIFI WLAN_VENDOR_SILABS X86_SGX X86_SGX_KVM X86_USER_SHADOW_STACK X86_X2APIC X86_X32_ABI XDP_SOCKETS XDP_SOCKETS_DIAG XFRM_ESPINTCP XFRM_INTERFACE XFRM_IPCOMP XFRM_MIGRATE XFRM_OFFLOAD XFRM_STATISTICS XFRM_SUB_POLICY XFRM_USER_COMPAT XFS_FS XFS_POSIX_ACL XFS_QUOTA XFS_RT XOR_BLOCKS YENTA YENTA_ENE_TUNE YENTA_O2 YENTA_RICOH YENTA_TI YENTA_TOSHIBA ZEROPLUS_FF ZLIB_DEFLATE ZONEFS_FS ZONE_DEVICE ZPOOL ZRAM ZRAM_DEF_COMP_LZORLE ZSMALLOC ZSTD_COMPRESS ZSWAP ZSWAP_COMPRESSOR_DEFAULT_LZO ZSWAP_DEFAULT_ON ZSWAP_SHRINKER_DEFAULT_ON ZSWAP_ZPOOL_DEFAULT_ZSMALLOC] disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed picked [v6.8 v6.7 v6.6 v6.4 v6.2 v6.0 v5.18 v5.16 v5.13 v5.10 v5.7 v5.4 v5.1 v4.19] out of 31 release tags testing release v6.8 testing commit e8f897f4afef0031fe618a8e94127a0934896aba gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: abd52ce4f2f2b9a83bb6b6a34e2905529053afabdb272b20d202b130a057ec69 all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] testing release v6.7 testing commit 0dd3ee31125508cd67f7e7172247f05b7fd1753a gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: cb039e10b488b27c0ecb07387882432b36f8ee2a67e039a5fec9f389cd72507b all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] testing release v6.6 testing commit ffc253263a1375a65fa6c9f62a893e9767fbebfa gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 4a95c41b47d022ae87135494cf167084464dad688c9f1a101e55d3ecaadff1d4 all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] testing release v6.4 testing commit 6995e2de6891c724bfeb2db33d7b87775f913ad1 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: e33aaee0ce0593f26308985985911b510a392c55ed7692b560150acc885440d8 all runs: OK false negative chance: 0.000 # git bisect start ffc253263a1375a65fa6c9f62a893e9767fbebfa 6995e2de6891c724bfeb2db33d7b87775f913ad1 Bisecting: 14974 revisions left to test after this (roughly 14 steps) [692f5510159c79bfa312a4e27a15e266232bfb4c] Merge tag 'asoc-v6.6' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus testing commit 692f5510159c79bfa312a4e27a15e266232bfb4c gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ea4f4e35ef45a1b4b1560604b352b3871133ff3b34901397b36d7541d62ba879 all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] # git bisect bad 692f5510159c79bfa312a4e27a15e266232bfb4c Bisecting: 7678 revisions left to test after this (roughly 13 steps) [b775d6c5859affe00527cbe74263de05cfe6b9f9] Merge tag 'mips_6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux testing commit b775d6c5859affe00527cbe74263de05cfe6b9f9 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ac5d6d120184807ef84a3aa1b66fb6a9fd0c36437a0bc8f01bafee063146b75e all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] # git bisect bad b775d6c5859affe00527cbe74263de05cfe6b9f9 Bisecting: 3061 revisions left to test after this (roughly 12 steps) [3a8a670eeeaa40d87bd38a587438952741980c18] Merge tag 'net-next-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next testing commit 3a8a670eeeaa40d87bd38a587438952741980c18 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: bc1afdfbc1046a6d71e4e4a426905fbec349499fcceb577bcfdacbd6048fad19 run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in corrupted run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: rcu detected stall in sys_bpf run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: INFO: rcu detected stall in corrupted run #9: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] # git bisect bad 3a8a670eeeaa40d87bd38a587438952741980c18 Bisecting: 2103 revisions left to test after this (roughly 11 steps) [6e17c6de3ddf3073741d9c91a796ee696914d8a0] Merge tag 'mm-stable-2023-06-24-19-15' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm testing commit 6e17c6de3ddf3073741d9c91a796ee696914d8a0 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8a5e53d9ece17a4960d1399c5b4e2cdc700a6514208eebff972811a3ad085380 all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] # git bisect bad 6e17c6de3ddf3073741d9c91a796ee696914d8a0 Bisecting: 1048 revisions left to test after this (roughly 10 steps) [2605e80d3438c77190f55b821c6575048c68268e] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux testing commit 2605e80d3438c77190f55b821c6575048c68268e gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8bec3e6445d81eb4e4fb9871cec9f9c2ad1221ee1d7f8f6d32b150c307d6a90b all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] # git bisect bad 2605e80d3438c77190f55b821c6575048c68268e Bisecting: 665 revisions left to test after this (roughly 9 steps) [0aa69d53ac7c30f6184f88f2e310d808b32b35a5] Merge tag 'for-6.5/io_uring-2023-06-23' of git://git.kernel.dk/linux testing commit 0aa69d53ac7c30f6184f88f2e310d808b32b35a5 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: e2ec5bfbb4f45b09edb7e8eb9e6d69569b9624b6615453b84d3be97e3e3c33fc all runs: OK false negative chance: 0.000 # git bisect good 0aa69d53ac7c30f6184f88f2e310d808b32b35a5 Bisecting: 341 revisions left to test after this (roughly 8 steps) [7cffdbe3607a6cc2dc02d135e13732ec36bc4e28] Merge tag 'x86-boot-2023-06-26' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 7cffdbe3607a6cc2dc02d135e13732ec36bc4e28 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d0f09f32da265581bb562986bb2dfcc1d742f490e13206c6b08db625a8e5be1f all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] # git bisect bad 7cffdbe3607a6cc2dc02d135e13732ec36bc4e28 Bisecting: 161 revisions left to test after this (roughly 7 steps) [658afed19ceed54a52b9e9e69c0791c8868ff55d] mtd: block: use a simple bool to track open for write testing commit 658afed19ceed54a52b9e9e69c0791c8868ff55d gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a9142d40d4cd8f382271d35ed8d785d6e4819f592fedd7a36f9514634f329bbe all runs: OK false negative chance: 0.000 # git bisect good 658afed19ceed54a52b9e9e69c0791c8868ff55d Bisecting: 80 revisions left to test after this (roughly 6 steps) [0b24be4691c9e6ea13ca70050d42a9f9032fa788] splice: don't call file_accessed in copy_splice_read testing commit 0b24be4691c9e6ea13ca70050d42a9f9032fa788 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 70f1e1a3ce84f03bc9b3e6bcd344a6799d943928604abd0f660353d64f19a344 all runs: OK false negative chance: 0.000 # git bisect good 0b24be4691c9e6ea13ca70050d42a9f9032fa788 Bisecting: 48 revisions left to test after this (roughly 5 steps) [f121ab7f4ac32ed2aa51035534926f9507a8308b] Merge tag 'irqchip-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms into irq/core testing commit f121ab7f4ac32ed2aa51035534926f9507a8308b gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3fbd6f4f41da2596ab48f3092931868f224d2285d83ed6ce7aa8a5abcaff6f38 all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] # git bisect bad f121ab7f4ac32ed2aa51035534926f9507a8308b Bisecting: 15 revisions left to test after this (roughly 4 steps) [e0b78e9fa34102ce82ef03c71eb13f0b79954ba3] Merge branch irq/loongarch-fixes-6.5 into irq/irqchip-next testing commit e0b78e9fa34102ce82ef03c71eb13f0b79954ba3 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d06e17b8fc19cb31d5b5b1e29a58f47b88c08235be17f6980bb47b9acd9ab9c8 all runs: OK false negative chance: 0.000 # git bisect good e0b78e9fa34102ce82ef03c71eb13f0b79954ba3 Bisecting: 7 revisions left to test after this (roughly 3 steps) [48f31e496488a25f443c0df52464da446fb1d10c] irqchip/stm32-exti: Fix warning on initialized field overwritten testing commit 48f31e496488a25f443c0df52464da446fb1d10c gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 862c47b496cfdd41d5906f3e8d9eb75b3689db6b43f8b1eee93c8373cb0fc88e all runs: OK false negative chance: 0.000 # git bisect good 48f31e496488a25f443c0df52464da446fb1d10c Bisecting: 3 revisions left to test after this (roughly 2 steps) [721255b9826bd11c7a38b585905fc2dd0fb94e52] genirq: Use a maple tree for interrupt descriptor management testing commit 721255b9826bd11c7a38b585905fc2dd0fb94e52 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 5b1bdd3a19bbf6fe17f5be725b7fbec5a7185b76ed096d3154e6601357e20a18 all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] # git bisect bad 721255b9826bd11c7a38b585905fc2dd0fb94e52 Bisecting: 1 revision left to test after this (roughly 1 step) [bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db] genirq: Use hlist for managing resend handlers testing commit bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 1c054048426054083bead38447abac5e9b0b43f01dfee010d3ee44ed17088094 all runs: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] # git bisect bad bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db Bisecting: 0 revisions left to test after this (roughly 0 steps) [d15121be7485655129101f3960ae6add40204463] Revert "softirq: Let ksoftirqd do its job" testing commit d15121be7485655129101f3960ae6add40204463 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 60b8674d8d12d3902f25311ebc42b078ffa7f3af3a2b93c600a0c4773c10ca81 run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in corrupted run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: rcu detected stall in sys_bpf run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: INFO: rcu detected stall in corrupted run #9: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] # git bisect bad d15121be7485655129101f3960ae6add40204463 d15121be7485655129101f3960ae6add40204463 is the first bad commit commit d15121be7485655129101f3960ae6add40204463 Author: Paolo Abeni Date: Mon May 8 08:17:44 2023 +0200 Revert "softirq: Let ksoftirqd do its job" This reverts the following commits: 4cd13c21b207 ("softirq: Let ksoftirqd do its job") 3c53776e29f8 ("Mark HI and TASKLET softirq synchronous") 1342d8080f61 ("softirq: Don't skip softirq execution when softirq thread is parking") in a single change to avoid known bad intermediate states introduced by a patch series reverting them individually. Due to the mentioned commit, when the ksoftirqd threads take charge of softirq processing, the system can experience high latencies. In the past a few workarounds have been implemented for specific side-effects of the initial ksoftirqd enforcement commit: commit 1ff688209e2e ("watchdog: core: make sure the watchdog_worker is not deferred") commit 8d5755b3f77b ("watchdog: softdog: fire watchdog even if softirqs do not get to run") commit 217f69743681 ("net: busy-poll: allow preemption in sk_busy_loop()") commit 3c53776e29f8 ("Mark HI and TASKLET softirq synchronous") But the latency problem still exists in real-life workloads, see the link below. The reverted commit intended to solve a live-lock scenario that can now be addressed with the NAPI threaded mode, introduced with commit 29863d41bb6e ("net: implement threaded-able napi poll loop support"), which is nowadays in a pretty stable status. While a complete solution to put softirq processing under nice resource control would be preferable, that has proven to be a very hard task. In the short term, remove the main pain point, and also simplify a bit the current softirq implementation. Signed-off-by: Paolo Abeni Signed-off-by: Thomas Gleixner Tested-by: Jason Xing Reviewed-by: Jakub Kicinski Reviewed-by: Eric Dumazet Reviewed-by: Sebastian Andrzej Siewior Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: netdev@vger.kernel.org Link: https://lore.kernel.org/netdev/305d7742212cbe98621b16be782b0562f1012cb6.camel@redhat.com Link: https://lore.kernel.org/r/57e66b364f1b6f09c9bc0316742c3b14f4ce83bd.1683526542.git.pabeni@redhat.com kernel/softirq.c | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-) accumulated error probability: 0.00 parent commit ac9a78681b921877518763ba0e89202254349d1b wasn't tested testing commit ac9a78681b921877518763ba0e89202254349d1b gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: e20a4819483a06e968bb3a0235933e98ebd33066dd309e7e651cb9cf23ff7ea8 culprit signature: 60b8674d8d12d3902f25311ebc42b078ffa7f3af3a2b93c600a0c4773c10ca81 parent signature: e20a4819483a06e968bb3a0235933e98ebd33066dd309e7e651cb9cf23ff7ea8 revisions tested: 26, total time: 6h17m12.83836734s (build: 2h44m11.914151047s, test: 3h11m41.674490741s) first bad commit: d15121be7485655129101f3960ae6add40204463 Revert "softirq: Let ksoftirqd do its job" recipients (to): ["bigeasy@linutronix.de" "edumazet@google.com" "kerneljasonxing@gmail.com" "kuba@kernel.org" "pabeni@redhat.com" "tglx@linutronix.de"] recipients (cc): [] crash: INFO: rcu detected stall in corrupted rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: rcu: (detected by 0, t=10002 jiffies, g=2197, q=478 ncpus=2) rcu: All QSes seen, last rcu_preempt kthread activity 10003 (4294951985-4294941982), jiffies_till_next_fqs=1, root ->qsmask 0x0 rcu: rcu_preempt kthread starved for 10004 jiffies! g2197 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. rcu: RCU grace-period kthread stack dump: task:rcu_preempt state:R running task stack:14736 pid:16 ppid:2 flags:0x00004000 Call Trace: context_switch kernel/sched/core.c:5343 [inline] __schedule+0x3aa/0xa30 kernel/sched/core.c:6669 schedule+0x59/0xb0 kernel/sched/core.c:6745 schedule_timeout+0xc2/0x150 kernel/time/timer.c:2167 rcu_gp_fqs_loop+0x13c/0x4a0 kernel/rcu/tree.c:1609 rcu_gp_kthread+0xc4/0x160 kernel/rcu/tree.c:1808 kthread+0xde/0x110 kernel/kthread.c:379 ret_from_fork+0x29/0x50 arch/x86/entry/entry_64.S:308 rcu: Stack dump where RCU GP kthread last ran: Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 CPU: 1 PID: 4042 Comm: syz-executor.2 Not tainted 6.4.0-rc1-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024 RIP: 0010:check_preemption_disabled+0x2/0xe0 lib/smp_processor_id.c:13 Code: 90 90 90 90 90 90 90 90 66 0f 1f 00 e9 07 fe ff ff 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 41 54 <55> 53 48 83 ec 08 65 8b 1d cd 19 53 7d 65 8b 05 c2 19 53 7d a9 ff RSP: 0018:ffffc900000d0e50 EFLAGS: 00000087 RAX: 0000000000000001 RBX: ffff888237d31028 RCX: 0000000029c5f01b RDX: ffff888237d30ff8 RSI: ffffffff836de75f RDI: ffffffff836cc66f RBP: ffff888237d31050 R08: 00000000000805a8 R09: ffff888114bf0a28 R10: 0000000000080000 R11: 0000000000000001 R12: 0000000000000001 R13: ffffffff82633424 R14: ffff888237d30e90 R15: ffffc900000d0f50 FS: 00007ff9025656c0(0000) GS:ffff888237d00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ff9019a8000 CR3: 000000016c8c4000 CR4: 00000000003506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: rcu_dynticks_curr_cpu_in_eqs include/linux/context_tracking.h:122 [inline] rcu_is_watching+0x11/0x50 kernel/rcu/tree.c:695 trace_lock_release include/trace/events/lock.h:69 [inline] lock_release+0x1e2/0x230 kernel/locking/lockdep.c:5702 __raw_spin_unlock_irq include/linux/spinlock_api_smp.h:157 [inline] _raw_spin_unlock_irq+0x16/0x50 kernel/locking/spinlock.c:202 spin_unlock_irq include/linux/spinlock.h:400 [inline] rps_unlock_irq_enable net/core/dev.c:247 [inline] process_backlog+0x154/0x170 net/core/dev.c:5957 __napi_poll+0x23/0x1d0 net/core/dev.c:6496 napi_poll net/core/dev.c:6563 [inline] net_rx_action+0x2ae/0x390 net/core/dev.c:6696 __do_softirq+0x13d/0x328 kernel/softirq.c:553 do_softirq kernel/softirq.c:454 [inline] do_softirq+0x7a/0xa0 kernel/softirq.c:441 __local_bh_enable_ip+0x60/0x70 kernel/softirq.c:381 local_bh_enable include/linux/bottom_half.h:33 [inline] bpf_test_run+0xf8/0x2d0 net/bpf/test_run.c:423 bpf_prog_test_run_skb+0x372/0x720 net/bpf/test_run.c:1229 bpf_prog_test_run kernel/bpf/syscall.c:3681 [inline] __sys_bpf+0xa18/0x2490 kernel/bpf/syscall.c:5076 __do_sys_bpf kernel/bpf/syscall.c:5162 [inline] __se_sys_bpf kernel/bpf/syscall.c:5160 [inline] __x64_sys_bpf+0x19/0x30 kernel/bpf/syscall.c:5160 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x3c/0xc0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x73/0xdd RIP: 0033:0x7ff90187dca9 Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ff9025650c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 00007ff9019abf80 RCX: 00007ff90187dca9 RDX: 0000000000000050 RSI: 0000000020000080 RDI: 000000000000000a RBP: 00007ff9018c947e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 000000000000000b R14: 00007ff9019abf80 R15: 00007ffc4317f158