ci starts bisection 2025-01-25 06:34:01.02430658 +0000 UTC m=+29838.943896107 bisecting cause commit starting from 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2 building syzkaller on 444551c40edde85b5e4900ae27d432bd1da7079c fetch other tags and check if the commit is present ensuring issue is reproducible on original commit 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2 testing commit 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 36d3563afe5230a494032bd9625ef06fee6f4cec604b389b35bff06559f6f2e4 run #0: crashed: INFO: rcu detected stall in sys_newfstatat run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: task hung in corrupted run #3: crashed: INFO: task hung in switchdev_deferred_process_work run #4: crashed: INFO: rcu detected stall in corrupted run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: crashed: INFO: task hung in caif_exit_net run #13: crashed: INFO: task hung in addrconf_dad_work run #14: OK run #15: crashed: INFO: task hung in corrupted run #16: OK run #17: OK run #18: OK run #19: crashed: INFO: task hung in linkwatch_event representative crash: INFO: rcu detected stall in sys_newfstatat, types: [HANG] check whether we can drop unnecessary instrumentation disabling configs for [ATOMIC_SLEEP LEAK UBSAN BUG KASAN LOCKDEP], they are not needed testing commit 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: cceef49478e0f01cae32db3ca2dd6e2f4d2a4a46994240177cff6b99be5b7128 run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: task hung in corrupted run #2: crashed: INFO: task hung in corrupted run #3: crashed: INFO: task hung in corrupted run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: task hung in corrupted run #6: crashed: INFO: rcu detected stall in corrupted run #7: crashed: INFO: task hung in corrupted run #8: crashed: INFO: task hung in corrupted run #9: crashed: INFO: task hung in corrupted run #10: crashed: INFO: task hung in corrupted run #11: crashed: INFO: task hung in corrupted run #12: crashed: INFO: task hung in corrupted run #13: crashed: INFO: rcu detected stall in corrupted run #14: crashed: INFO: task hung in corrupted run #15: crashed: INFO: task hung in corrupted run #16: crashed: INFO: task hung in del_device_store run #17: crashed: INFO: task hung in corrupted run #18: crashed: INFO: task hung in corrupted run #19: crashed: INFO: rcu detected stall in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] the bug reproduces without the instrumentation disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP LEAK UBSAN BUG], they are not needed kconfig minimization: base=4045 full=8253 leaves diff=2128 split chunks (needed=false): <2128> split chunk #0 of len 2128 into 5 parts testing without sub-chunk 1/5 disabling configs for [LOCKDEP ATOMIC_SLEEP LEAK UBSAN BUG KASAN], they are not needed testing commit 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 0e29fee2b38051fc1de80b9f57f071a66408249520c4ae993d78681d3182b353 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in corrupted run #2: crashed: INFO: task hung in corrupted run #3: crashed: INFO: task hung in corrupted run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in corrupted run #6: crashed: INFO: task hung in corrupted run #7: crashed: INFO: task hung in corrupted run #8: crashed: INFO: task hung in corrupted run #9: crashed: INFO: task hung in corrupted run #10: crashed: INFO: task hung in corrupted run #11: crashed: INFO: task hung in corrupted run #12: crashed: INFO: rcu detected stall in corrupted run #13: crashed: INFO: task hung in corrupted run #14: crashed: INFO: rcu detected stall in corrupted run #15: crashed: INFO: task hung in wg_destruct run #16: crashed: INFO: task hung in corrupted run #17: crashed: INFO: rcu detected stall in corrupted run #18: crashed: INFO: task hung in addrconf_dad_work run #19: crashed: no output from test machine representative crash: INFO: task hung in del_device_store, types: [HANG] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 failed building 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2: drivers/gpu/drm/bridge/aux-bridge.c:116: undefined reference to `devm_drm_of_get_bridge' testing without sub-chunk 3/5 disabling configs for [UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP LEAK], they are not needed testing commit 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8d1f171f1f2aeafeb68151891f7a38c65789b5ceaf5c96b12b5fdf66be2398c7 all runs: OK false negative chance: 0.000 testing without sub-chunk 4/5 disabling configs for [ATOMIC_SLEEP LEAK UBSAN BUG KASAN LOCKDEP], they are not needed testing commit 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 071b6c00e5f4bd45eaaddb148fce05acaafdb5d1b1a3e308b75e0772e3d5d227 run #0: crashed: INFO: task hung in linkwatch_event run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in nsim_destroy run #7: crashed: INFO: task hung in del_device_store run #8: crashed: INFO: rcu detected stall in corrupted run #9: crashed: INFO: task hung in rtnl_newlink run #10: crashed: INFO: task hung in crda_timeout_work run #11: crashed: INFO: task hung in netdev_run_todo run #12: crashed: INFO: task hung in addrconf_dad_work run #13: crashed: INFO: task hung in rtnetlink_rcv_msg run #14: crashed: no output from test machine run #15: crashed: INFO: task hung in switchdev_deferred_process_work run #16: crashed: INFO: task hung in linkwatch_event run #17: crashed: no output from test machine run #18: OK run #19: OK representative crash: INFO: task hung in linkwatch_event, types: [HANG] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP LEAK], they are not needed testing commit 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3f94550d8b9d657cb84b1bd37a8baaeebcbc8e6d82e6bdfafec1fa4cf7d22529 all runs: OK false negative chance: 0.000 minimized to 1276 configs; suspects: [6LOWPAN ARCH_ENABLE_MEMORY_HOTREMOVE ASUS_WMI ATM AX25 BCMA BLK_DEV_ZONED BPF_SYSCALL BRIDGE BRIDGE_NETFILTER CAN CARDBUS CFG80211 CFG80211_WEXT CHARGER_BQ24190 CHECKPOINT_RESTORE CMA COMMON_CLK CONTIG_ALLOC DAX DLM DRM DRM_BRIDGE DRM_GUD DRM_I915 DRM_I915_CAPTURE_ERROR DRM_I915_COMPRESS_ERROR DRM_I915_USERPTR DRM_KMS_HELPER DRM_MIPI_DSI DRM_PANEL DRM_PANEL_BRIDGE DRM_PANEL_EDP DRM_PANEL_ORIENTATION_QUIRKS DRM_SIMPLEDRM DRM_TTM DRM_TTM_HELPER DRM_UDL DRM_VGEM DRM_VIRTIO_GPU DRM_VIRTIO_GPU_KMS DRM_VKMS DRM_VMWGFX DUMMY DVB_AF9013 DVB_AF9033 DVB_AS102 DVB_AS102_FE DVB_B2C2_FLEXCOP DVB_B2C2_FLEXCOP_USB DVB_CORE DVB_DIB3000MB DVB_DIB3000MC DVB_EC100 DVB_GP8PSK_FE DVB_RTL2830 DVB_RTL2832 DVB_RTL2832_SDR DVB_TEST_DRIVERS DVB_TTUSB_BUDGET DVB_TTUSB_DEC DVB_USB DVB_USB_A800 DVB_USB_AF9005 DVB_USB_AF9005_REMOTE DVB_USB_AF9015 DVB_USB_AF9035 DVB_USB_ANYSEE DVB_USB_AU6610 DVB_USB_AZ6007 DVB_USB_AZ6027 DVB_USB_CE6230 DVB_USB_CINERGY_T2 DVB_USB_CXUSB DVB_USB_CXUSB_ANALOG DVB_USB_DIB0700 DVB_USB_DIB3000MC DVB_USB_DIBUSB_MB DVB_USB_DIBUSB_MC DVB_USB_DIGITV DVB_USB_DTT200U DVB_USB_DTV5100 DVB_USB_DVBSKY DVB_USB_DW2102 DVB_USB_EC168 DVB_USB_GL861 DVB_USB_GP8PSK DVB_USB_LME2510 DVB_USB_M920X DVB_USB_MXL111SF DVB_USB_NOVA_T_USB2 DVB_USB_OPERA1 DVB_USB_PCTV452E DVB_USB_RTL28XXU DVB_USB_TECHNISAT_USB2 DVB_USB_TTUSB2 DVB_USB_UMT_010 DVB_USB_V2 DVB_USB_VP702X DVB_USB_VP7045 DVB_USB_ZD1301 DVB_VIDTV DVB_ZL10353 ECRYPT_FS ECRYPT_FS_MESSAGING EDAC EEPROM_93CX6 EFS_FS ENCRYPTED_KEYS EQUALIZER EROFS_FS EROFS_FS_BACKED_BY_FILE EROFS_FS_POSIX_ACL EROFS_FS_SECURITY EROFS_FS_XATTR EROFS_FS_ZIP EVM EVM_ADD_XATTRS EVM_ATTR_FSUUID EXFAT_FS EXPORTFS_BLOCK_OPS EXT3_FS EXT3_FS_POSIX_ACL EXT3_FS_SECURITY EXTCON EXTCON_INTEL_CHT_WC EXTCON_PTN5150 EXTCON_USBC_TUSB320 F2FS_CHECK_FS F2FS_FAULT_INJECTION F2FS_FS F2FS_FS_COMPRESSION F2FS_FS_LZ4 F2FS_FS_LZ4HC F2FS_FS_LZO F2FS_FS_LZORLE F2FS_FS_POSIX_ACL F2FS_FS_SECURITY F2FS_FS_XATTR F2FS_FS_ZSTD F2FS_STAT_FS FANOTIFY FANOTIFY_ACCESS_PERMISSIONS FB FB_CFB_COPYAREA FB_CFB_FILLRECT FB_CFB_IMAGEBLIT FB_CORE FB_DEFERRED_IO FB_DEVICE FB_IOMEM_FOPS FB_IOMEM_HELPERS FB_NOTIFY FB_SYSMEM_FOPS FB_SYSMEM_HELPERS FB_SYSMEM_HELPERS_DEFERRED FB_SYS_COPYAREA FB_SYS_FILLRECT FB_SYS_IMAGEBLIT FB_TILEBLITTING FB_VESA FB_VGA16 FB_VIRTUAL FDDI FIREWIRE FIREWIRE_NET FIREWIRE_OHCI FIREWIRE_SBP2 FONT_8x16 FONT_8x8 FONT_SUPPORT FRAMEBUFFER_CONSOLE FRAMEBUFFER_CONSOLE_DETECT_PRIMARY FRAMEBUFFER_CONSOLE_ROTATION FS_DAX FS_DAX_PMD FS_ENCRYPTION FS_ENCRYPTION_ALGS FS_STACK FS_VERITY FS_VERITY_BUILTIN_SIGNATURES FTL FUSE_DAX FUSE_FS FW_LOADER_COMPRESS FW_LOADER_PAGED_BUF FW_LOADER_SYSFS FW_LOADER_USER_HELPER FW_LOADER_USER_HELPER_FALLBACK GACT_PROB GARP GENERIC_PHY GET_FREE_REGION GFS2_FS GFS2_FS_LOCKING_DLM GNSS GNSS_USB GOOGLE_COREBOOT_TABLE GOOGLE_FIRMWARE GOOGLE_MEMCONSOLE GOOGLE_MEMCONSOLE_COREBOOT GOOGLE_VPD GPIOLIB GPIOLIB_IRQCHIP GPIO_ACPI GPIO_DLN2 GPIO_LJCA GPIO_VIPERBOARD GREENASIA_FF GREYBUS GREYBUS_BRIDGED_PHY GREYBUS_ES2 GREYBUS_HID GREYBUS_USB GROUP_SCHED_WEIGHT GTP GUEST_PERF_EVENTS GVE HAMRADIO HAVE_ARCH_NODE_DEV_GROUP HAVE_ARCH_USERFAULTFD_MINOR HAVE_ARCH_USERFAULTFD_WP HAVE_BOOTMEM_INFO_NODE HAVE_CLK_PREPARE HAVE_KVM_CPU_RELAX_INTERCEPT HAVE_KVM_DIRTY_RING HAVE_KVM_DIRTY_RING_ACQ_REL HAVE_KVM_DIRTY_RING_TSO HAVE_KVM_IRQCHIP HAVE_KVM_IRQ_BYPASS HAVE_KVM_IRQ_ROUTING HAVE_KVM_MSI HAVE_KVM_NO_POLL HAVE_KVM_PFNCACHE HAVE_KVM_PM_NOTIFIER HAVE_KVM_READONLY_MEM HAVE_SCHED_AVG_IRQ HDLC HDLC_CISCO HDLC_FR HDLC_PPP HDLC_RAW HDLC_RAW_ETH HDLC_X25 HDMI HFSPLUS_FS HFS_FS HID_ACCUTOUCH HID_ACRUX HID_ACRUX_FF HID_ALPS HID_APPLEIR HID_ASUS HID_AUREAL HID_BATTERY_STRENGTH HID_BETOP_FF HID_BIGBEN_FF HID_CMEDIA HID_CORSAIR HID_COUGAR HID_CP2112 HID_CREATIVE_SB0540 HID_ELAN HID_ELECOM HID_ELO HID_EMS_FF HID_EVISION HID_FT260 HID_GEMBIRD HID_GFRM HID_GLORIOUS HID_GOOGLE_STADIA_FF HID_GREENASIA HID_GT683R HID_HOLTEK HID_ICADE HID_JABRA HID_KEYTOUCH HID_KYE HID_LCPOWER HID_LED HID_LENOVO HID_LETSKETCH HID_LOGITECH_DJ HID_LOGITECH_HIDPP HID_MACALLY HID_MAGICMOUSE HID_MALTRON HID_MAYFLASH HID_MCP2200 HID_MCP2221 HID_MEGAWORLD_FF HID_MULTITOUCH HID_NINTENDO HID_NTI HID_ORTEK HID_PENMOUNT HID_PICOLCD HID_PICOLCD_BACKLIGHT HID_PICOLCD_CIR HID_PICOLCD_FB HID_PICOLCD_LCD HID_PICOLCD_LEDS HID_PLANTRONICS HID_PRIMAX HID_PRODIKEYS HID_PXRC HID_RAZER HID_RETRODE HID_RMI HID_ROCCAT HID_SAITEK HID_SEMITEK HID_SENSOR_ACCEL_3D HID_SENSOR_ALS HID_SENSOR_CUSTOM_INTEL_HINGE HID_SENSOR_CUSTOM_SENSOR HID_SENSOR_DEVICE_ROTATION HID_SENSOR_GYRO_3D HID_SENSOR_HUB HID_SENSOR_HUMIDITY HID_SENSOR_IIO_COMMON HID_SENSOR_IIO_TRIGGER HID_SENSOR_INCLINOMETER_3D HID_SENSOR_MAGNETOMETER_3D HID_SENSOR_PRESS HID_SENSOR_PROX HID_SENSOR_TEMP HID_SIGMAMICRO HID_SPEEDLINK HID_STEELSERIES HID_THINGM HID_TIVO HID_TOPRE HID_TWINHAN HID_U2FZERO HID_UCLOGIC HID_UDRAW_PS3 HID_VIEWSONIC HID_VIVALDI HID_VIVALDI_COMMON HID_VRC2 HID_WACOM HID_WALTOP HID_WIIMOTE HID_XIAOMI HID_XINMO HID_ZEROPLUS HID_ZYDACRON HMM_MIRROR HOLTEK_FF HOTPLUG_PCI_PCIE HPET_MMAP HPET_MMAP_DEFAULT HPFS_FS HSR I2C_ALGOBIT I2C_CHARDEV I2C_CP2615 I2C_DESIGNWARE_CORE I2C_DESIGNWARE_PLATFORM I2C_DIOLAN_U2C I2C_DLN2 I2C_HID_ACPI I2C_HID_CORE I2C_HID_OF I2C_LJCA I2C_MUX I2C_MUX_REG I2C_ROBOTFUZZ_OSIF I2C_SI4713 I2C_SLAVE I2C_SLAVE_EEPROM I2C_TINY_USB I2C_VIPERBOARD IEEE802154 IEEE802154_6LOWPAN IEEE802154_ATUSB IEEE802154_DRIVERS IEEE802154_HWSIM IEEE802154_NL802154_EXPERIMENTAL IEEE802154_SOCKET IFB IIO IIO_BUFFER IIO_KFIFO_BUF IIO_TRIGGER IIO_TRIGGERED_BUFFER IKCONFIG IKCONFIG_PROC IMA IMA_APPRAISE IMA_APPRAISE_MODSIG IMA_DEFAULT_HASH_SHA256 IMA_LSM_RULES IMA_MEASURE_ASYMMETRIC_KEYS IMA_NG_TEMPLATE IMA_QUEUE_EARLY_BOOT_KEYS IMA_READ_POLICY IMA_WRITE_POLICY INET6_ESPINTCP INET6_ESP_OFFLOAD INET6_IPCOMP INET6_TUNNEL INET6_XFRM_TUNNEL INET_AH INET_DCCP_DIAG INET_DIAG INET_DIAG_DESTROY INET_ESP INET_ESPINTCP INET_ESP_OFFLOAD INET_IPCOMP INET_MPTCP_DIAG INET_RAW_DIAG INET_SCTP_DIAG INET_TCP_DIAG INET_UDP_DIAG INET_XFRM_TUNNEL INFINIBAND INFINIBAND_ADDR_TRANS INFINIBAND_ADDR_TRANS_CONFIGFS INFINIBAND_IPOIB INFINIBAND_IPOIB_CM INFINIBAND_IPOIB_DEBUG INFINIBAND_ISER INFINIBAND_ON_DEMAND_PAGING INFINIBAND_RTRS INFINIBAND_SRP INFINIBAND_USER_ACCESS INFINIBAND_USER_MAD INFINIBAND_USER_MEM INPUT_ATI_REMOTE2 INPUT_CM109 INPUT_IMS_PCU INPUT_JOYDEV INPUT_JOYSTICK INPUT_KEYSPAN_REMOTE INPUT_LEDS INPUT_MOUSE INPUT_MOUSEDEV INPUT_MOUSEDEV_PSAUX INPUT_POWERMATE INPUT_UINPUT INPUT_YEALINK INTEGRITY INTEGRITY_ASYMMETRIC_KEYS INTEGRITY_AUDIT INTEGRITY_SIGNATURE INTEGRITY_TRUSTED_KEYRING INTEL_CHTWC_INT33FE INTEL_IDMA64 INTEL_IOATDMA INTEL_IOMMU_DEFAULT_ON INTEL_IOMMU_SVM INTEL_ISHTP_ECLITE INTEL_ISH_FIRMWARE_DOWNLOADER INTEL_ISH_HID INTEL_SOC_PMIC_CHTWC INTERVAL_TREE_SPAN_ITER IOMMUFD IOMMUFD_DRIVER IOMMUFD_DRIVER_CORE IOMMUFD_TEST IP6_NF_MATCH_AH IP6_NF_MATCH_EUI64 IP6_NF_MATCH_FRAG IP6_NF_MATCH_HL IP6_NF_MATCH_MH IP6_NF_MATCH_OPTS IP6_NF_MATCH_RPFILTER IP6_NF_MATCH_RT IP6_NF_MATCH_SRH IP6_NF_NAT IP6_NF_RAW IP6_NF_SECURITY IP6_NF_TARGET_HL IP6_NF_TARGET_MASQUERADE IP6_NF_TARGET_NPT IP6_NF_TARGET_SYNPROXY IPV6_FOU IPV6_FOU_TUNNEL IPV6_GRE IPV6_ILA IPV6_MIP6 IPV6_MROUTE IPV6_MROUTE_MULTIPLE_TABLES IPV6_MULTIPLE_TABLES IPV6_OPTIMISTIC_DAD IPV6_PIMSM_V2 IPV6_ROUTER_PREF IPV6_ROUTE_INFO IPV6_RPL_LWTUNNEL IPV6_SEG6_BPF IPV6_SEG6_HMAC IPV6_SEG6_LWTUNNEL IPV6_SIT_6RD IPV6_SUBTREES IPV6_TUNNEL IPV6_VTI IPVLAN IPVLAN_L3S IPVTAP IP_DCCP IP_DCCP_CCID3 IP_DCCP_TFRC_LIB IP_FIB_TRIE_STATS IP_MROUTE_MULTIPLE_TABLES IP_NF_ARPFILTER IP_NF_ARPTABLES IP_NF_ARP_MANGLE IP_NF_MATCH_AH IP_NF_MATCH_ECN IP_NF_MATCH_RPFILTER IP_NF_MATCH_TTL IP_NF_RAW IP_NF_SECURITY IP_NF_TARGET_ECN IP_NF_TARGET_NETMAP IP_NF_TARGET_REDIRECT IP_NF_TARGET_SYNPROXY IP_NF_TARGET_TTL IP_ROUTE_CLASSID IP_SCTP IP_SET IP_SET_BITMAP_IP IP_SET_BITMAP_IPMAC IP_SET_BITMAP_PORT IP_SET_HASH_IP IP_SET_HASH_IPMAC IP_SET_HASH_IPMARK IP_SET_HASH_IPPORT IP_SET_HASH_IPPORTIP IP_SET_HASH_IPPORTNET IP_SET_HASH_MAC IP_SET_HASH_NET IP_SET_HASH_NETIFACE IP_SET_HASH_NETNET IP_SET_HASH_NETPORT IP_SET_HASH_NETPORTNET IP_SET_LIST_SET IP_VS IP_VS_DH IP_VS_FO IP_VS_FTP IP_VS_IPV6 IP_VS_LBLC IP_VS_LBLCR IP_VS_LC IP_VS_MH IP_VS_NFCT IP_VS_NQ IP_VS_OVF IP_VS_PE_SIP IP_VS_PROTO_AH IP_VS_PROTO_AH_ESP IP_VS_PROTO_ESP IP_VS_PROTO_SCTP IP_VS_PROTO_TCP IP_VS_PROTO_UDP IP_VS_RR IP_VS_SED IP_VS_SH IP_VS_TWOS IP_VS_WLC IP_VS_WRR IRQ_BYPASS_MANAGER IRQ_POLL IRQ_REMAP IRQ_TIME_ACCOUNTING IR_IGORPLUGUSB IR_IGUANA IR_IMON IR_IMON_RAW IR_MCEUSB IR_REDRAT3 IR_STREAMZAP IR_TOY IR_TTUSBIR ISDN ISDN_CAPI JFFS2_CMODE_PRIORITY JFFS2_COMPRESSION_OPTIONS JFFS2_FS JFFS2_FS_POSIX_ACL JFFS2_FS_SECURITY JFFS2_FS_WRITEBUFFER JFFS2_FS_XATTR JFFS2_LZO JFFS2_RTIME JFFS2_RUBIN JFFS2_SUMMARY JFFS2_ZLIB JFS_DEBUG JFS_FS JFS_POSIX_ACL JFS_SECURITY JOYSTICK_IFORCE JOYSTICK_IFORCE_USB JOYSTICK_PXRC JOYSTICK_XPAD JOYSTICK_XPAD_FF JOYSTICK_XPAD_LEDS KARMA_PARTITION KCOV KCOV_ENABLE_COMPARISONS KCOV_INSTRUMENT_ALL KEYS_REQUEST_CACHE KEY_DH_OPERATIONS KEY_NOTIFICATIONS KSM KVM KVM_AMD KVM_ASYNC_PF KVM_COMMON KVM_COMPAT KVM_ELIDE_TLB_FLUSH_IF_YOUNG KVM_GENERIC_DIRTYLOG_READ_PROTECT KVM_GENERIC_HARDWARE_ENABLING KVM_GENERIC_MEMORY_ATTRIBUTES KVM_GENERIC_MMU_NOTIFIER KVM_GENERIC_PRE_FAULT_MEMORY KVM_GENERIC_PRIVATE_MEM KVM_HYPERV KVM_INTEL KVM_MMIO KVM_PRIVATE_MEM KVM_PROVE_MMU KVM_SW_PROTECTED_VM KVM_VFIO KVM_X86 KVM_XEN KVM_XFER_TO_GUEST_WORK L2TP L2TP_ETH L2TP_IP L2TP_V3 LAPB LAPBETHER LCD_CLASS_DEVICE LDM_PARTITION LEGACY_PTYS LIBCRC32C LIBNVDIMM LINEAR_RANGES LLC LLC2 LOGIG940_FF LOGIRUMBLEPAD2_FF LOGO LOGO_LINUX_MONO LOGO_LINUX_VGA16 LPC_ICH LWTUNNEL LWTUNNEL_BPF LZ4HC_COMPRESS LZ4_COMPRESS MAC80211 MAC80211_HAS_RC MAC80211_HWSIM MAC80211_MESH MAC80211_RC_DEFAULT_MINSTREL MAC80211_RC_MINSTREL MAC802154 MACSEC MACVLAN MACVTAP MAC_PARTITION MAPPING_DIRTY_HELPERS MDIO_MVUSB MD_RAID0 MD_RAID1 MD_RAID10 MD_RAID456 MEDIA_ANALOG_TV_SUPPORT MEDIA_ATTACH MEDIA_CAMERA_SUPPORT MEDIA_CEC_SUPPORT MEDIA_CONTROLLER MEDIA_CONTROLLER_DVB MEDIA_DIGITAL_TV_SUPPORT MEDIA_RADIO_SUPPORT MEDIA_SDR_SUPPORT MEDIA_SUPPORT MEDIA_SUPPORT_FILTER MEDIA_TEST_SUPPORT MEDIA_TUNER MEDIA_TUNER_MSI001 MEDIA_USB_SUPPORT MEMORY_BALLOON MEMORY_HOTPLUG MEMORY_HOTPLUG_DEFAULT_ONLINE MEMORY_HOTREMOVE MEMORY_ISOLATION MEMREGION MEMSTICK MEMSTICK_REALTEK_USB MEM_SOFT_DIRTY MFD_CORE MFD_DLN2 MFD_SYSCON MFD_VIPERBOARD MHI_BUS MHI_WWAN_CTRL MHP_MEMMAP_ON_MEMORY MICROCHIP_PHY MINIX_FS MINIX_SUBPARTITION MIN_HEAP MISC_RTSX MISC_RTSX_USB MISDN MISDN_DSP MISDN_HFCUSB MISDN_L1OIP MKISS MLX4_CORE MLX4_INFINIBAND MMC MMC_REALTEK_USB MMC_USHC MMC_VUB300 MODULE_SRCVERSION_ALL MODVERSIONS MOST MOST_USB_HDM MOUSE_APPLETOUCH MOUSE_BCM5974 MOUSE_PS2 MOUSE_PS2_ALPS MOUSE_PS2_BYD MOUSE_PS2_CYPRESS MOUSE_PS2_FOCALTECH MOUSE_PS2_LIFEBOOK MOUSE_PS2_LOGIPS2PP MOUSE_PS2_SMBUS MOUSE_PS2_SYNAPTICS MOUSE_PS2_SYNAPTICS_SMBUS MOUSE_PS2_TRACKPOINT MOUSE_SYNAPTICS_USB MPLS MPLS_IPTUNNEL MPLS_ROUTING MPTCP MPTCP_IPV6 MRP MTD MTD_BLKDEVS MTD_BLOCK MTD_BLOCK2MTD MTD_CFI_I1 MTD_CFI_I2 MTD_MAP_BANK_WIDTH_1 MTD_MAP_BANK_WIDTH_2 MTD_MAP_BANK_WIDTH_4 MTD_MTDRAM MTD_PHRAM MTD_SLRAM MUSB_PIO_ONLY ND_BTT ND_CLAIM ND_PFN NETDEVSIM NETFILTER_ADVANCED NETFILTER_BPF_LINK NETFILTER_FAMILY_ARP NETFILTER_FAMILY_BRIDGE NETFILTER_NETLINK_ACCT NETFILTER_NETLINK_GLUE_CT NETFILTER_NETLINK_OSF NETFILTER_NETLINK_QUEUE NETFILTER_SYNPROXY NETFILTER_XTABLES_COMPAT NETFILTER_XT_CONNMARK NETFILTER_XT_MATCH_BPF NETFILTER_XT_MATCH_CGROUP NETFILTER_XT_MATCH_CLUSTER NETFILTER_XT_MATCH_COMMENT NETFILTER_XT_MATCH_CONNBYTES NETFILTER_XT_MATCH_CONNLABEL NETFILTER_XT_MATCH_CONNLIMIT NETFILTER_XT_MATCH_CONNMARK NETFILTER_XT_MATCH_CPU NETFILTER_XT_MATCH_DCCP NETFILTER_XT_MATCH_DEVGROUP NETFILTER_XT_MATCH_DSCP NETFILTER_XT_MATCH_ECN NETFILTER_XT_MATCH_ESP NETFILTER_XT_MATCH_HASHLIMIT NETFILTER_XT_MATCH_HELPER NETFILTER_XT_MATCH_HL NETFILTER_XT_MATCH_IPCOMP NETFILTER_XT_MATCH_IPRANGE NETFILTER_XT_MATCH_IPVS NETFILTER_XT_MATCH_L2TP NETFILTER_XT_MATCH_LENGTH NETFILTER_XT_MATCH_LIMIT NETFILTER_XT_MATCH_MAC NETFILTER_XT_MATCH_MARK NETFILTER_XT_MATCH_MULTIPORT NETFILTER_XT_MATCH_NFACCT NETFILTER_XT_MATCH_OSF NETFILTER_XT_MATCH_OWNER NETFILTER_XT_MATCH_PHYSDEV NETFILTER_XT_MATCH_PKTTYPE NETFILTER_XT_MATCH_QUOTA NETFILTER_XT_MATCH_RATEEST NETFILTER_XT_MATCH_REALM NETFILTER_XT_MATCH_RECENT NETFILTER_XT_MATCH_SCTP NETFILTER_XT_MATCH_SOCKET NETFILTER_XT_MATCH_STATISTIC NETFILTER_XT_MATCH_STRING NETFILTER_XT_MATCH_TCPMSS NETFILTER_XT_MATCH_TIME NETFILTER_XT_MATCH_U32 NETFILTER_XT_SET NETFILTER_XT_TARGET_AUDIT NETFILTER_XT_TARGET_CHECKSUM NETFILTER_XT_TARGET_CLASSIFY NETFILTER_XT_TARGET_CONNMARK NETFILTER_XT_TARGET_CT NETFILTER_XT_TARGET_DSCP NETFILTER_XT_TARGET_HL NETFILTER_XT_TARGET_HMARK NETFILTER_XT_TARGET_IDLETIMER NETFILTER_XT_TARGET_LED NETFILTER_XT_TARGET_MARK NETFILTER_XT_TARGET_NETMAP NETFILTER_XT_TARGET_NFQUEUE NETFILTER_XT_TARGET_NOTRACK NETFILTER_XT_TARGET_RATEEST NETFILTER_XT_TARGET_REDIRECT NETFILTER_XT_TARGET_TCPOPTSTRIP NETFILTER_XT_TARGET_TEE NETFILTER_XT_TARGET_TPROXY NETFILTER_XT_TARGET_TRACE NETLABEL NETLINK_DIAG NETROM NET_9P_RDMA NET_ACT_BPF NET_ACT_CONNMARK NET_ACT_CSUM NET_ACT_CT NET_ACT_CTINFO NET_ACT_GACT NET_ACT_GATE NET_ACT_IFE NET_ACT_MIRRED NET_ACT_MPLS NET_ACT_NAT NET_ACT_PEDIT NET_ACT_POLICE NET_ACT_SAMPLE NET_ACT_SIMP NET_ACT_SKBEDIT NET_ACT_SKBMOD NET_ACT_TUNNEL_KEY NET_ACT_VLAN NET_CLS_BASIC NET_CLS_BPF NET_CLS_FLOW NET_CLS_FLOWER NET_CLS_FW NET_CLS_MATCHALL NET_CLS_ROUTE4 NET_DEVLINK NET_DEVMEM NET_DROP_MONITOR NET_DSA NET_DSA_TAG_BRCM NET_DSA_TAG_BRCM_COMMON NET_DSA_TAG_BRCM_PREPEND NET_DSA_TAG_MTK NET_DSA_TAG_QCA NET_DSA_TAG_RTL4_A NET_EMATCH_CANID NET_EMATCH_CMP NET_EMATCH_IPSET NET_EMATCH_IPT NET_EMATCH_META NET_EMATCH_NBYTE NET_EMATCH_TEXT NET_EMATCH_U32 NET_FC NET_FOU NET_FOU_IP_TUNNELS NET_IFE NET_IFE_SKBMARK NET_IFE_SKBPRIO NET_IFE_SKBTCINDEX NET_IPGRE NET_IPGRE_BROADCAST NET_IPGRE_DEMUX NET_IPIP NET_IPVTI NET_KEY NET_KEY_MIGRATE NET_L3_MASTER_DEV NET_MPLS_GSO NET_NCSI NET_NSH NET_REDIRECT NET_SCH_CAKE NET_SCH_CBS NET_SCH_CHOKE NET_SCH_CODEL NET_SCH_DRR NET_SCH_ETF NET_SCH_ETS NET_SCH_FQ NET_SCH_FQ_CODEL NET_SCH_FQ_PIE NET_SCH_GRED NET_SCH_HFSC NET_SCH_HHF NET_SCH_HTB NET_SCH_INGRESS NET_SCH_MQPRIO NET_SCH_MQPRIO_LIB NET_SCH_MULTIQ NET_SCH_NETEM NET_SCH_PIE NET_SCH_PLUG NET_SCH_PRIO NET_SCH_QFQ NET_SCH_RED NET_SCH_SFB NET_SCH_SFQ NET_SCH_SKBPRIO NET_SCH_TAPRIO NET_SCH_TBF NET_SCH_TEQL NET_SHAPER NET_SOCK_MSG NET_SWITCHDEV NET_TC_SKB_EXT NET_TEAM NET_TEAM_MODE_ACTIVEBACKUP NET_TEAM_MODE_BROADCAST NET_TEAM_MODE_LOADBALANCE NET_TEAM_MODE_RANDOM NET_TEAM_MODE_ROUNDROBIN NET_UDP_TUNNEL NET_VENDOR_HISILICON NET_VRF NFC NFC_DIGITAL NFC_FDP NFC_HCI NFC_MRVL NFC_MRVL_USB NFC_NCI NFC_NCI_UART NFC_PN533 NFC_PN533_USB NFC_PORT100 NFC_SHDLC NFC_SIM NFC_VIRTUAL_NCI NFSD NFSD_BLOCKLAYOUT NFSD_FLEXFILELAYOUT NFSD_PNFS NFSD_SCSILAYOUT NFSD_V3_ACL NFSD_V4 NFSD_V4_2_INTER_SSC NFSD_V4_SECURITY_LABEL NFS_FSCACHE NFS_V4_1 NFS_V4_2 NFS_V4_2_READ_PLUS NFS_V4_2_SSC_HELPER NFS_V4_SECURITY_LABEL NFT_BRIDGE_META NFT_BRIDGE_REJECT NFT_COMPAT NFT_COMPAT_ARP NFT_CONNLIMIT NFT_CT NFT_DUP_IPV4 NFT_DUP_IPV6 NFT_DUP_NETDEV NFT_FIB NFT_FIB_INET NFT_FIB_IPV4 NFT_FIB_IPV6 NFT_FIB_NETDEV NFT_FLOW_OFFLOAD NFT_FWD_NETDEV NFT_HASH NFT_LIMIT NFT_LOG NFT_MASQ NFT_NAT NFT_NUMGEN NFT_OSF NFT_QUEUE NFT_QUOTA NFT_REDIR NFT_REJECT NFT_REJECT_INET NFT_REJECT_IPV4 NFT_REJECT_IPV6 NFT_REJECT_NETDEV NFT_SOCKET NFT_SYNPROXY NFT_TPROXY NFT_TUNNEL NFT_XFRM NF_CONNTRACK_AMANDA NF_CONNTRACK_BRIDGE NF_CONNTRACK_BROADCAST NF_CONNTRACK_EVENTS NF_CONNTRACK_H323 NF_CONNTRACK_LABELS NF_CONNTRACK_MARK NF_CONNTRACK_NETBIOS_NS NF_CONNTRACK_OVS NF_CONNTRACK_PPTP NF_CONNTRACK_SANE NF_CONNTRACK_SNMP NF_CONNTRACK_TFTP NF_CONNTRACK_TIMEOUT NF_CONNTRACK_TIMESTAMP NF_CONNTRACK_ZONES NF_CT_NETLINK_HELPER NF_CT_NETLINK_TIMEOUT NF_CT_PROTO_DCCP NF_CT_PROTO_GRE NF_CT_PROTO_SCTP NF_CT_PROTO_UDPLITE NF_DUP_IPV4 NF_DUP_IPV6 NF_DUP_NETDEV NF_FLOW_TABLE NF_FLOW_TABLE_INET NF_NAT_AMANDA NF_NAT_H323 NF_NAT_OVS NF_NAT_PPTP NF_NAT_REDIRECT NF_NAT_SNMP_BASIC NF_NAT_TFTP NF_SOCKET_IPV4 NF_SOCKET_IPV6 NF_TABLES NF_TABLES_ARP NF_TABLES_BRIDGE NF_TABLES_INET NF_TABLES_IPV4 NF_TABLES_IPV6 NF_TABLES_NETDEV NF_TPROXY_IPV4 NF_TPROXY_IPV6 NILFS2_FS NINTENDO_FF NLMON NLS_CODEPAGE_1250 NLS_CODEPAGE_1251 NLS_CODEPAGE_737 NOP_USB_XCEIV PAGE_POOL PARPORT PARTITION_ADVANCED PCCARD PCMCIA PHONET PSAMPLE RADIO_ADAPTERS RADIO_SI470X RADIO_SI4713 RAS RC_CORE RC_DEVICES REGULATOR RFKILL SND SOUND SPI SSB STAGING TAP TARGET_CORE TRANSPARENT_HUGEPAGE TUN TYPEC TYPEC_MUX_PI3USB30532 USB_AMD5536UDC USB_ATM USB_CDNS3 USB_CDNS3_HOST USB_CDNS3_PCI_WRAP USB_CDNSP_GADGET USB_CDNSP_HOST USB_CDNSP_PCI USB_CDNS_HOST USB_CDNS_SUPPORT USB_CHAOSKEY USB_CHIPIDEA USB_CHIPIDEA_GENERIC USB_CHIPIDEA_HOST USB_CHIPIDEA_MSM USB_CHIPIDEA_NPCM USB_CHIPIDEA_PCI USB_CHIPIDEA_UDC USB_CONFIGFS USB_CONFIGFS_ACM USB_CONFIGFS_ECM USB_CONFIGFS_ECM_SUBSET USB_CONFIGFS_EEM USB_CONFIGFS_F_FS USB_CONFIGFS_F_HID USB_CONFIGFS_F_LB_SS USB_CONFIGFS_F_MIDI USB_CONFIGFS_F_MIDI2 USB_CONFIGFS_F_PRINTER USB_CONFIGFS_F_TCM USB_CONFIGFS_F_UAC1 USB_CONFIGFS_F_UAC1_LEGACY USB_CONFIGFS_F_UAC2 USB_CONFIGFS_F_UVC USB_CONFIGFS_MASS_STORAGE USB_CONFIGFS_NCM USB_CONFIGFS_OBEX USB_CONFIGFS_PHONET USB_CONFIGFS_RNDIS USB_CONFIGFS_SERIAL USB_CONN_GPIO USB_CXACRU USB_CYPRESS_CY7C63 USB_CYTHERM USB_DSBR USB_DUMMY_HCD USB_DWC2 USB_DWC2_HOST USB_DWC2_PCI USB_DWC3 USB_DWC3_GADGET USB_DWC3_HAPS USB_DWC3_OF_SIMPLE USB_DWC3_PCI USB_DWC3_ULPI USB_DYNAMIC_MINORS USB_EG20T USB_EHCI_FSL USB_EHCI_HCD_PLATFORM USB_EHCI_ROOT_HUB_TT USB_EHSET_TEST_FIXTURE USB_EMI26 USB_EMI62 USB_EPSON2888 USB_EZUSB_FX2 USB_FEW_INIT_RETRIES USB_F_ACM USB_F_ECM USB_F_EEM USB_F_FS USB_F_HID USB_F_MASS_STORAGE USB_F_MIDI USB_F_MIDI2 USB_F_NCM USB_F_OBEX USB_F_PHONET USB_F_PRINTER USB_F_RNDIS USB_F_SERIAL USB_F_SS_LB USB_F_SUBSET USB_F_TCM USB_F_UAC1 USB_F_UAC1_LEGACY USB_F_UAC2 USB_F_UVC USB_GADGET USB_GADGETFS USB_GADGET_DEBUG_FILES USB_GADGET_DEBUG_FS USB_GL860 USB_GOKU USB_GR_UDC USB_GSPCA USB_GSPCA_BENQ USB_GSPCA_CONEX USB_GSPCA_CPIA1 USB_GSPCA_DTCS033 USB_GSPCA_ETOMS USB_GSPCA_FINEPIX USB_GSPCA_JEILINJ USB_GSPCA_JL2005BCD USB_GSPCA_KINECT USB_GSPCA_KONICA USB_GSPCA_MARS USB_GSPCA_MR97310A USB_GSPCA_NW80X USB_GSPCA_OV519 USB_GSPCA_OV534 USB_GSPCA_OV534_9 USB_GSPCA_PAC207 USB_GSPCA_PAC7302 USB_GSPCA_PAC7311 USB_GSPCA_SE401 USB_GSPCA_SN9C2028 USB_GSPCA_SN9C20X USB_GSPCA_SONIXB USB_GSPCA_SONIXJ USB_GSPCA_SPCA1528 USB_GSPCA_SPCA500 USB_GSPCA_SPCA501 USB_GSPCA_SPCA505 USB_GSPCA_SPCA506 USB_GSPCA_SPCA508 USB_GSPCA_SPCA561 USB_GSPCA_SQ905 USB_GSPCA_SQ905C USB_GSPCA_SQ930X USB_GSPCA_STK014 USB_GSPCA_STK1135 USB_GSPCA_STV0680 USB_GSPCA_SUNPLUS USB_GSPCA_T613 USB_GSPCA_TOPRO USB_GSPCA_TOUPTEK USB_GSPCA_TV8532 USB_GSPCA_VC032X USB_GSPCA_VICAM USB_GSPCA_XIRLINK_CIT USB_GSPCA_ZC3XX USB_HACKRF USB_HCD_BCMA USB_HCD_SSB USB_HSIC_USB3503 USB_HSIC_USB4604 USB_HSO USB_HUB_USB251XB USB_IDMOUSE USB_IOWARRIOR USB_IPHETH USB_ISIGHTFW USB_ISP116X_HCD USB_ISP1301 USB_ISP1760 USB_ISP1760_DUAL_ROLE USB_ISP1760_HCD USB_ISP1761_UDC USB_KAWETH USB_KC2190 USB_KEENE USB_LAN78XX USB_LCD USB_LD USB_LEDS_TRIGGER_USBPORT USB_LED_TRIG USB_LEGOTOWER USB_LGM_PHY USB_LIBCOMPOSITE USB_LINK_LAYER_TEST USB_LJCA USB_M5602 USB_MA901 USB_MAX3420_UDC USB_MAX3421_HCD USB_MDC800 USB_MICROTEK USB_MR800 USB_MSI2500 USB_MUSB_DUAL_ROLE USB_MUSB_HDRC USB_MV_U3D USB_MV_UDC USB_NET2272 USB_NET2272_DMA USB_NET2280 USB_NET_AQC111 USB_NET_AX88179_178A USB_NET_AX8817X USB_NET_CDCETHER USB_NET_CDC_EEM USB_NET_CDC_MBIM USB_NET_CDC_NCM USB_NET_CDC_SUBSET USB_NET_CDC_SUBSET_ENABLE USB_NET_CH9200 USB_NET_CX82310_ETH USB_NET_DM9601 USB_NET_GL620A USB_NET_HUAWEI_CDC_NCM USB_NET_INT51X1 USB_NET_KALMIA USB_NET_MCS7830 USB_NET_NET1080 USB_NET_PLUSB USB_NET_QMI_WWAN USB_NET_RNDIS_HOST USB_NET_SMSC75XX USB_NET_SMSC95XX USB_NET_SR9700 USB_NET_SR9800 USB_NET_ZAURUS USB_OHCI_HCD_PLATFORM USB_OTG USB_OTG_FSM USB_OXU210HP_HCD USB_PEGASUS USB_PULSE8_CEC USB_PWC USB_PWC_INPUT_EVDEV USB_PXA27X USB_R8A66597 USB_R8A66597_HCD USB_RAINSHADOW_CEC USB_RAREMONO USB_RAW_GADGET USB_ROLES_INTEL_XHCI USB_ROLE_SWITCH USB_RTL8150 USB_RTL8152 USB_RTL8153_ECM USB_S2255 USB_SERIAL USB_SERIAL_AIRCABLE USB_SERIAL_ARK3116 USB_SERIAL_BELKIN USB_SERIAL_CH341 USB_SERIAL_CONSOLE USB_SERIAL_CP210X USB_SERIAL_CYBERJACK USB_SERIAL_CYPRESS_M8 USB_SERIAL_DEBUG USB_SERIAL_DIGI_ACCELEPORT USB_SERIAL_EDGEPORT USB_SERIAL_EDGEPORT_TI USB_SERIAL_EMPEG USB_SERIAL_F81232 USB_SERIAL_F8153X USB_SERIAL_FTDI_SIO USB_SERIAL_GARMIN USB_SERIAL_GENERIC USB_SERIAL_IPAQ USB_SERIAL_IPW USB_SERIAL_IR USB_SERIAL_IUU USB_SERIAL_KEYSPAN USB_SERIAL_KEYSPAN_PDA USB_SERIAL_KLSI USB_SERIAL_KOBIL_SCT USB_SERIAL_MCT_U232 USB_SERIAL_METRO USB_SERIAL_MOS7715_PARPORT USB_SERIAL_MOS7720 USB_SERIAL_MOS7840 USB_SERIAL_MXUPORT USB_SERIAL_NAVMAN USB_SERIAL_OMNINET USB_SERIAL_OPTICON USB_SERIAL_OPTION USB_SERIAL_OTI6858 USB_SERIAL_PL2303 USB_SERIAL_QCAUX USB_SERIAL_QT2 USB_SERIAL_QUALCOMM USB_SERIAL_SAFE USB_SERIAL_SIERRAWIRELESS USB_SERIAL_SIMPLE USB_SERIAL_SPCP8X5 USB_SERIAL_SSU100 USB_SERIAL_SYMBOL USB_SERIAL_TI USB_SERIAL_UPD78F0730 USB_SERIAL_VISOR USB_SERIAL_WHITEHEAT USB_SERIAL_WISHBONE USB_SERIAL_WWAN USB_SERIAL_XR USB_SERIAL_XSENS_MT USB_SEVSEG USB_SI470X USB_SI4713 USB_SIERRA_NET USB_SISUSBVGA USB_SL811_CS USB_SL811_HCD USB_SL811_HCD_ISO USB_SNP_CORE USB_SPEEDTOUCH USB_STORAGE_ALAUDA USB_STORAGE_CYPRESS_ATACB USB_STORAGE_DATAFAB USB_STORAGE_ENE_UB6250 USB_STORAGE_FREECOM USB_STORAGE_ISD200 USB_STORAGE_JUMPSHOT USB_STORAGE_KARMA USB_STORAGE_ONETOUCH USB_STORAGE_SDDR09 USB_STORAGE_SDDR55 USB_STORAGE_USBAT USB_STV06XX USB_TEST USB_TMC USB_TRANCEVIBRATOR USB_UAS USB_UEAGLEATM USB_ULPI_BUS USB_USBNET USB_USS720 USB_U_AUDIO USB_U_ETHER USB_U_SERIAL USB_VIDEO_CLASS USB_VIDEO_CLASS_INPUT_EVDEV USB_VL600 USB_WDM USB_XHCI_DBGCAP USB_XHCI_PCI_RENESAS USB_XHCI_PLATFORM USB_XUSBATM USB_YUREX USERFAULTFD USERIO USERMODE_DRIVER USER_RETURN_NOTIFIER UVC_COMMON U_SERIAL_CONSOLE V4L2_MEM2MEM_DEV V4L_TEST_DRIVERS VALIDATE_FS_PARSER VDPA VDPA_SIM VDPA_SIM_BLOCK VDPA_SIM_NET VETH VFIO VFIO_DEVICE_CDEV VFIO_PCI VFIO_PCI_CORE VFIO_PCI_INTX VFIO_PCI_MMAP VFIO_VIRQFD VGASTATE VHOST VHOST_CROSS_ENDIAN_LEGACY VHOST_IOTLB VHOST_NET VHOST_RING VHOST_TASK VHOST_VDPA VHOST_VSOCK VIDEO VIDEOBUF2_CORE VIDEOBUF2_DMA_CONTIG VIDEOBUF2_DMA_SG VIDEOBUF2_MEMOPS VIDEOBUF2_V4L2 VIDEOBUF2_VMALLOC VIDEOMODE_HELPERS VIDEO_AU0828 VIDEO_AU0828_RC VIDEO_AU0828_V4L2 VIDEO_CS53L32A VIDEO_CX231XX VIDEO_CX231XX_ALSA VIDEO_CX231XX_DVB VIDEO_CX231XX_RC VIDEO_CX2341X VIDEO_CX25840 VIDEO_DEV VIDEO_EM28XX VIDEO_EM28XX_ALSA VIDEO_EM28XX_DVB VIDEO_EM28XX_RC VIDEO_EM28XX_V4L2 VIDEO_GO7007 VIDEO_GO7007_LOADER VIDEO_GO7007_USB VIDEO_GO7007_USB_S2250_BOARD VIDEO_HDPVR VIDEO_MSP3400 VIDEO_PVRUSB2 VIDEO_PVRUSB2_DVB VIDEO_PVRUSB2_SYSFS VIDEO_SAA711X VIDEO_STK1160 VIDEO_TUNER VIDEO_TVEEPROM VIDEO_USBTV VIDEO_V4L2_I2C VIDEO_V4L2_SUBDEV_API VIDEO_V4L2_TPG VIDEO_VICODEC VIDEO_VIM2M VIDEO_VIMC VIDEO_VIVID VIDEO_VIVID_CEC VIDEO_WM8775 VIPERBOARD_ADC VIRTIO_BALLOON VIRTIO_DMA_SHARED_BUFFER VIRTIO_FS VIRTIO_MEM VIRTIO_MMIO VIRTIO_MMIO_CMDLINE_DEVICES VIRTIO_PMEM VIRTIO_VDPA VIRTIO_VSOCKETS VIRTIO_VSOCKETS_COMMON VIRT_WIFI VLAN_8021Q VLAN_8021Q_GVRP VLAN_8021Q_MVRP VMAP_PFN VMWARE_VMCI VMXNET3 VP_VDPA VSOCKETS VSOCKETS_DIAG VSOCKETS_LOOPBACK VSOCKMON VT_HW_CONSOLE_BINDING VXFS_FS WAN WANT_DEV_COREDUMP WATCH_QUEUE WEXT_CORE WEXT_PROC WIREGUARD WIRELESS WLAN WLAN_VENDOR_ADMTEK WLAN_VENDOR_SILABS WWAN X25 X86_SGX X86_SGX_KVM X86_USER_SHADOW_STACK X86_X2APIC X86_X32_ABI XDP_SOCKETS XDP_SOCKETS_DIAG XFRM_ESPINTCP XFRM_INTERFACE XFRM_IPCOMP XFRM_MIGRATE XFRM_OFFLOAD XFRM_STATISTICS XFRM_SUB_POLICY XFRM_USER_COMPAT XFS_FS XFS_POSIX_ACL XFS_QUOTA XFS_RT XILLYBUS_CLASS XILLYUSB XOR_BLOCKS YENTA YENTA_ENE_TUNE YENTA_O2 YENTA_RICOH YENTA_TI YENTA_TOSHIBA Z3FOLD Z3FOLD_DEPRECATED ZEROPLUS_FF ZLIB_DEFLATE ZONEFS_FS ZONE_DEVICE ZPOOL ZRAM ZRAM_BACKEND_FORCE_LZO ZRAM_BACKEND_LZO ZRAM_DEF_COMP_LZO ZSMALLOC ZSTD_COMPRESS ZSWAP ZSWAP_COMPRESSOR_DEFAULT_842 ZSWAP_DEFAULT_ON ZSWAP_SHRINKER_DEFAULT_ON ZSWAP_ZPOOL_DEFAULT_Z3FOLD_DEPRECATED] disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed picked [v6.12 v6.11 v6.10 v6.8 v6.6 v6.4 v6.2 v6.0 v5.17 v5.14 v5.11 v5.8 v5.5 v5.2 v4.20 v4.19] out of 35 release tags testing release v6.12 testing commit adc218676eef25575469234709c2d87185ca223a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 4e9ec4379fd3535c1a497ab06d8c290c04fea5cbc099b12926b44b83003edce2 run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: task hung in del_device_store run #7: crashed: INFO: task hung in del_device_store run #8: crashed: INFO: rcu detected stall in corrupted run #9: crashed: INFO: task hung in addrconf_dad_work run #10: crashed: INFO: task hung in reg_todo run #11: crashed: INFO: rcu detected stall in corrupted run #12: crashed: INFO: task hung in addrconf_dad_work run #13: crashed: INFO: task hung in rtnetlink_rcv_msg run #14: crashed: INFO: task hung in addrconf_dad_work run #15: crashed: INFO: rcu detected stall in corrupted run #16: crashed: INFO: task hung in linkwatch_event run #17: crashed: INFO: task hung in linkwatch_event run #18: crashed: no output from test machine run #19: OK representative crash: INFO: rcu detected stall in corrupted, types: [HANG] testing release v6.11 testing commit 98f7e32f20d28ec452afb208f9cffc08448a2652 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 41e77817d54f13fa99b62a455a0b5dfa72dba5c55e0895bb25c3bf18a0ee4e56 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in new_device_store run #2: crashed: INFO: task hung in new_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in reg_todo run #7: crashed: INFO: task hung in addrconf_dad_work run #8: crashed: no output from test machine run #9: crashed: no output from test machine run #10: crashed: no output from test machine run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in del_device_store, types: [HANG] testing release v6.10 testing commit 0c3836482481200ead7b416ca80c68a29cfdaabd gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ed4de8bf67f689f46c19f3ed19e9059e6223a4c82632d02e33e41048a38bcfd4 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in new_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: INFO: task hung in new_device_store run #8: crashed: INFO: task hung in del_device_store run #9: crashed: INFO: task hung in rtnetlink_rcv_msg run #10: crashed: no output from test machine run #11: crashed: no output from test machine run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in del_device_store, types: [HANG] testing release v6.8 testing commit e8f897f4afef0031fe618a8e94127a0934896aba gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: f4927251db7005f03ca70b2aa6d2397637900da3b8b6d5287c02cfa40792c9c8 run #0: ignore: lost connection to test machine run #1: crashed: INFO: task hung in new_device_store run #2: crashed: INFO: task hung in new_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in new_device_store run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in devlink_pernet_pre_exit run #7: crashed: INFO: task hung in del_device_store run #8: crashed: INFO: task hung in del_device_store run #9: crashed: INFO: task hung in del_device_store run #10: crashed: INFO: task hung in tun_chr_close run #11: crashed: INFO: task hung in reg_check_chans_work run #12: crashed: no output from test machine run #13: crashed: no output from test machine run #14: OK run #15: crashed: no output from test machine run #16: crashed: no output from test machine run #17: OK run #18: crashed: no output from test machine run #19: OK representative crash: INFO: task hung in new_device_store, types: [HANG UNKNOWN] testing release v6.6 testing commit ffc253263a1375a65fa6c9f62a893e9767fbebfa gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 60ab9d48b40bcd9073fd70127b139b6a6a704696d29fd7d6861ad15ecb5ffe94 run #0: crashed: INFO: task hung in new_device_store run #1: crashed: INFO: task hung in new_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in new_device_store run #4: crashed: INFO: task hung in devlink_pernet_pre_exit run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: INFO: task hung in del_device_store run #8: crashed: INFO: task hung in new_device_store run #9: crashed: INFO: task hung in nsim_drv_remove run #10: crashed: no output from test machine run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in new_device_store, types: [HANG] testing release v6.4 testing commit 6995e2de6891c724bfeb2db33d7b87775f913ad1 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ce6e249ad35dad3752335b72b56299bc6fa5db668080e63c0607c01b99df7c4b all runs: OK false negative chance: 0.000 # git bisect start ffc253263a1375a65fa6c9f62a893e9767fbebfa 6995e2de6891c724bfeb2db33d7b87775f913ad1 Bisecting: 14974 revisions left to test after this (roughly 14 steps) [692f5510159c79bfa312a4e27a15e266232bfb4c] Merge tag 'asoc-v6.6' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus testing commit 692f5510159c79bfa312a4e27a15e266232bfb4c gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2ef5dbd527140b075c5950571c748986a06a56373479abb86def2f3108b655b8 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: INFO: task hung in del_device_store run #8: crashed: INFO: task hung in del_device_store run #9: crashed: INFO: task hung in nsim_drv_remove run #10: crashed: INFO: task hung in del_device_store run #11: crashed: INFO: task hung in del_device_store run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad 692f5510159c79bfa312a4e27a15e266232bfb4c Bisecting: 7678 revisions left to test after this (roughly 13 steps) [b775d6c5859affe00527cbe74263de05cfe6b9f9] Merge tag 'mips_6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux testing commit b775d6c5859affe00527cbe74263de05cfe6b9f9 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3674a1483047fc4190467e919800a80a1dcdc17bcbaea55e741e64f2439c857b run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: INFO: task hung in del_device_store run #8: crashed: INFO: task hung in del_device_store run #9: OK run #10: OK run #11: OK run #12: OK run #13: crashed: INFO: task hung in reg_todo run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad b775d6c5859affe00527cbe74263de05cfe6b9f9 Bisecting: 3061 revisions left to test after this (roughly 12 steps) [3a8a670eeeaa40d87bd38a587438952741980c18] Merge tag 'net-next-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next testing commit 3a8a670eeeaa40d87bd38a587438952741980c18 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 0dfe41ea23cd015945193c2f032f34ca6f5df20314d7d21c119686c634ffa83a run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in new_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in del_device_store run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: crashed: no output from test machine representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad 3a8a670eeeaa40d87bd38a587438952741980c18 Bisecting: 2103 revisions left to test after this (roughly 11 steps) [6e17c6de3ddf3073741d9c91a796ee696914d8a0] Merge tag 'mm-stable-2023-06-24-19-15' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm testing commit 6e17c6de3ddf3073741d9c91a796ee696914d8a0 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 87557f34e9fddff3a40b477bb9709cf37b93041e29bfc0b9f51539d70fc966bb run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in nsim_drv_remove run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in nsim_drv_remove run #7: crashed: INFO: task hung in del_device_store run #8: crashed: INFO: task hung in del_device_store run #9: crashed: INFO: task hung in del_device_store run #10: crashed: INFO: task hung in del_device_store run #11: crashed: no output from test machine run #12: OK run #13: OK run #14: OK run #15: crashed: no output from test machine run #16: OK run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad 6e17c6de3ddf3073741d9c91a796ee696914d8a0 Bisecting: 1048 revisions left to test after this (roughly 10 steps) [2605e80d3438c77190f55b821c6575048c68268e] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux testing commit 2605e80d3438c77190f55b821c6575048c68268e gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: cb9d23ae100c5ea70212b45de35aaa8fa7a4f86893e204ac8d88167e4f6f7521 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: no output from test machine run #8: crashed: no output from test machine run #9: crashed: no output from test machine run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: crashed: INFO: task hung in switchdev_deferred_process_work run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad 2605e80d3438c77190f55b821c6575048c68268e Bisecting: 665 revisions left to test after this (roughly 9 steps) [0aa69d53ac7c30f6184f88f2e310d808b32b35a5] Merge tag 'for-6.5/io_uring-2023-06-23' of git://git.kernel.dk/linux testing commit 0aa69d53ac7c30f6184f88f2e310d808b32b35a5 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 075530180629eb3e1fbc5ee628efa6b59571306b146b90786c794cbbde6cb77e all runs: OK false negative chance: 0.000 # git bisect good 0aa69d53ac7c30f6184f88f2e310d808b32b35a5 Bisecting: 341 revisions left to test after this (roughly 8 steps) [7cffdbe3607a6cc2dc02d135e13732ec36bc4e28] Merge tag 'x86-boot-2023-06-26' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 7cffdbe3607a6cc2dc02d135e13732ec36bc4e28 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8b6a80556441d24865cfe10474dae2e117053c6da0c60454e0f8f95f66e6777d run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: no output from test machine run #4: crashed: no output from test machine run #5: crashed: no output from test machine run #6: crashed: INFO: task hung in tun_chr_close run #7: OK run #8: OK run #9: crashed: no output from test machine run #10: crashed: no output from test machine run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: no output from test machine, types: [UNKNOWN] # git bisect bad 7cffdbe3607a6cc2dc02d135e13732ec36bc4e28 Bisecting: 161 revisions left to test after this (roughly 7 steps) [658afed19ceed54a52b9e9e69c0791c8868ff55d] mtd: block: use a simple bool to track open for write testing commit 658afed19ceed54a52b9e9e69c0791c8868ff55d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ceb1f8441c9adc0cf5309983beddd32af2930472e6f10029869bb80a33e64ba6 all runs: OK false negative chance: 0.000 # git bisect good 658afed19ceed54a52b9e9e69c0791c8868ff55d Bisecting: 80 revisions left to test after this (roughly 6 steps) [0b24be4691c9e6ea13ca70050d42a9f9032fa788] splice: don't call file_accessed in copy_splice_read testing commit 0b24be4691c9e6ea13ca70050d42a9f9032fa788 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 40d4ac9c91ab1f36b5ad1b5df8418f854896609e4988ac72cabe8c0a8c174c02 all runs: OK false negative chance: 0.000 # git bisect good 0b24be4691c9e6ea13ca70050d42a9f9032fa788 Bisecting: 48 revisions left to test after this (roughly 5 steps) [f121ab7f4ac32ed2aa51035534926f9507a8308b] Merge tag 'irqchip-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms into irq/core testing commit f121ab7f4ac32ed2aa51035534926f9507a8308b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 55644e36a9533ff7a67e7e8bd52b467c4456f62cbcbab83290734ad2a4662350 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in nsim_drv_remove run #5: crashed: INFO: task hung in nsim_drv_remove run #6: crashed: INFO: task hung in nsim_drv_remove run #7: crashed: INFO: task hung in del_device_store run #8: crashed: no output from test machine run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: crashed: no output from test machine run #18: OK run #19: OK representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad f121ab7f4ac32ed2aa51035534926f9507a8308b Bisecting: 15 revisions left to test after this (roughly 4 steps) [e0b78e9fa34102ce82ef03c71eb13f0b79954ba3] Merge branch irq/loongarch-fixes-6.5 into irq/irqchip-next testing commit e0b78e9fa34102ce82ef03c71eb13f0b79954ba3 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 60d9c3d78c63dbef5b4672c7f812f5a1fc68bce98220b318292567757f36b7b4 all runs: OK false negative chance: 0.000 # git bisect good e0b78e9fa34102ce82ef03c71eb13f0b79954ba3 Bisecting: 7 revisions left to test after this (roughly 3 steps) [48f31e496488a25f443c0df52464da446fb1d10c] irqchip/stm32-exti: Fix warning on initialized field overwritten testing commit 48f31e496488a25f443c0df52464da446fb1d10c gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9b5ea63e41f124efeda180a2e44a759b45e0658d2555691ca1c919b7be43f1c9 all runs: OK false negative chance: 0.000 # git bisect good 48f31e496488a25f443c0df52464da446fb1d10c Bisecting: 3 revisions left to test after this (roughly 2 steps) [721255b9826bd11c7a38b585905fc2dd0fb94e52] genirq: Use a maple tree for interrupt descriptor management testing commit 721255b9826bd11c7a38b585905fc2dd0fb94e52 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9f3b0b70c80c989dd58aacd5cf109f1ebe31546cfc40c242f0d0e4cfb5703ec1 run #0: crashed: INFO: task hung in register_netdevice_notifier run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in devlink_free run #3: crashed: no output from test machine run #4: crashed: no output from test machine run #5: crashed: no output from test machine run #6: crashed: no output from test machine run #7: crashed: no output from test machine run #8: OK run #9: OK run #10: crashed: no output from test machine run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: no output from test machine, types: [UNKNOWN] # git bisect bad 721255b9826bd11c7a38b585905fc2dd0fb94e52 Bisecting: 1 revision left to test after this (roughly 1 step) [bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db] genirq: Use hlist for managing resend handlers testing commit bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 578de8cada3a8b18b42876ef60c8d6090f696d81b4079d5c2095d8cfe11ae0dd run #0: crashed: INFO: task hung in register_netdevice_notifier run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in devlink_free run #5: crashed: no output from test machine run #6: crashed: no output from test machine run #7: OK run #8: OK run #9: crashed: no output from test machine run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in register_netdevice_notifier, types: [HANG UNKNOWN] # git bisect bad bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db Bisecting: 0 revisions left to test after this (roughly 0 steps) [d15121be7485655129101f3960ae6add40204463] Revert "softirq: Let ksoftirqd do its job" testing commit d15121be7485655129101f3960ae6add40204463 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6f56df3b7b3c9fc52622672dd675601e96e8983ab5d1305de79cb4cc54a77b14 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in devlink_free run #4: crashed: no output from test machine run #5: crashed: no output from test machine run #6: crashed: no output from test machine run #7: OK run #8: OK run #9: crashed: no output from test machine run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: crashed: no output from test machine run #17: OK run #18: crashed: no output from test machine run #19: OK representative crash: no output from test machine, types: [UNKNOWN] # git bisect bad d15121be7485655129101f3960ae6add40204463 d15121be7485655129101f3960ae6add40204463 is the first bad commit commit d15121be7485655129101f3960ae6add40204463 Author: Paolo Abeni Date: Mon May 8 08:17:44 2023 +0200 Revert "softirq: Let ksoftirqd do its job" This reverts the following commits: 4cd13c21b207 ("softirq: Let ksoftirqd do its job") 3c53776e29f8 ("Mark HI and TASKLET softirq synchronous") 1342d8080f61 ("softirq: Don't skip softirq execution when softirq thread is parking") in a single change to avoid known bad intermediate states introduced by a patch series reverting them individually. Due to the mentioned commit, when the ksoftirqd threads take charge of softirq processing, the system can experience high latencies. In the past a few workarounds have been implemented for specific side-effects of the initial ksoftirqd enforcement commit: commit 1ff688209e2e ("watchdog: core: make sure the watchdog_worker is not deferred") commit 8d5755b3f77b ("watchdog: softdog: fire watchdog even if softirqs do not get to run") commit 217f69743681 ("net: busy-poll: allow preemption in sk_busy_loop()") commit 3c53776e29f8 ("Mark HI and TASKLET softirq synchronous") But the latency problem still exists in real-life workloads, see the link below. The reverted commit intended to solve a live-lock scenario that can now be addressed with the NAPI threaded mode, introduced with commit 29863d41bb6e ("net: implement threaded-able napi poll loop support"), which is nowadays in a pretty stable status. While a complete solution to put softirq processing under nice resource control would be preferable, that has proven to be a very hard task. In the short term, remove the main pain point, and also simplify a bit the current softirq implementation. Signed-off-by: Paolo Abeni Signed-off-by: Thomas Gleixner Tested-by: Jason Xing Reviewed-by: Jakub Kicinski Reviewed-by: Eric Dumazet Reviewed-by: Sebastian Andrzej Siewior Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: netdev@vger.kernel.org Link: https://lore.kernel.org/netdev/305d7742212cbe98621b16be782b0562f1012cb6.camel@redhat.com Link: https://lore.kernel.org/r/57e66b364f1b6f09c9bc0316742c3b14f4ce83bd.1683526542.git.pabeni@redhat.com kernel/softirq.c | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-) accumulated error probability: 0.00 parent commit ac9a78681b921877518763ba0e89202254349d1b wasn't tested testing commit ac9a78681b921877518763ba0e89202254349d1b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 40d553a659d37ac50d8330bc85ee6ab860ddf20b6d78d0ad990ca7fe4d95dbae culprit signature: 6f56df3b7b3c9fc52622672dd675601e96e8983ab5d1305de79cb4cc54a77b14 parent signature: 40d553a659d37ac50d8330bc85ee6ab860ddf20b6d78d0ad990ca7fe4d95dbae reproducer is flaky (0.43 repro chance estimate) revisions tested: 27, total time: 10h11m30.944446115s (build: 5h32m19.838984914s, test: 4h20m12.608164095s) first bad commit: d15121be7485655129101f3960ae6add40204463 Revert "softirq: Let ksoftirqd do its job" recipients (to): ["bigeasy@linutronix.de" "edumazet@google.com" "kerneljasonxing@gmail.com" "kuba@kernel.org" "pabeni@redhat.com" "tglx@linutronix.de"] recipients (cc): [] crash: no output from test machine