ci starts bisection 2024-11-24 22:44:20.679860967 +0000 UTC m=+195932.236897993 bisecting cause commit starting from cfba9f07a1d6aeca38f47f1f472cfb0ba133d341 building syzkaller on 68da6d951a345757b69b764ceb8dda1e9d65b038 ensuring issue is reproducible on original commit cfba9f07a1d6aeca38f47f1f472cfb0ba133d341 testing commit cfba9f07a1d6aeca38f47f1f472cfb0ba133d341 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: e9c0db37fd4cfb33ebb536894bb2d1fa07f4084bb5fe735b6d637509c72c37d0 run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: task hung in corrupted run #2: crashed: INFO: task hung in addrconf_dad_work run #3: crashed: INFO: task hung in linkwatch_event run #4: crashed: INFO: task hung in on_each_cpu_cond_mask run #5: crashed: INFO: task hung in corrupted run #6: crashed: INFO: task hung in addrconf_dad_work run #7: crashed: INFO: task hung in linkwatch_event run #8: crashed: INFO: task hung in corrupted run #9: crashed: INFO: rcu detected stall in corrupted run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: crashed: INFO: task hung in addrconf_dad_work run #17: OK run #18: OK run #19: OK representative crash: INFO: rcu detected stall in corrupted, types: [HANG] check whether we can drop unnecessary instrumentation disabling configs for [UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP LEAK], they are not needed testing commit cfba9f07a1d6aeca38f47f1f472cfb0ba133d341 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 857b46217f9b3d4bb312990454772a853eb84cd78b1e9ab495b653cbad627f40 all runs: crashed: INFO: task hung in corrupted representative crash: INFO: task hung in corrupted, types: [HANG] the bug reproduces without the instrumentation disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP LEAK UBSAN BUG], they are not needed kconfig minimization: base=4045 full=8239 leaves diff=2123 split chunks (needed=false): <2123> split chunk #0 of len 2123 into 5 parts testing without sub-chunk 1/5 disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit cfba9f07a1d6aeca38f47f1f472cfb0ba133d341 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6a741da0e1d5be69049b8709457ef986a513d92938894678723c104580a57682 run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: task hung in corrupted run #2: crashed: INFO: task hung in corrupted run #3: crashed: INFO: task hung in corrupted run #4: crashed: INFO: task hung in corrupted run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in corrupted run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: INFO: task hung in corrupted run #9: crashed: INFO: task hung in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit cfba9f07a1d6aeca38f47f1f472cfb0ba133d341 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 failed building cfba9f07a1d6aeca38f47f1f472cfb0ba133d341: drivers/gpu/drm/bridge/aux-bridge.c:116: undefined reference to `devm_drm_of_get_bridge' testing without sub-chunk 3/5 disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit cfba9f07a1d6aeca38f47f1f472cfb0ba133d341 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 34c254bfa77e8bb72b41fab56a9663315226de1285054e497ac8b35672a63816 all runs: OK false negative chance: 0.000 testing without sub-chunk 4/5 disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit cfba9f07a1d6aeca38f47f1f472cfb0ba133d341 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 92547b844214f0b56f9de46bbc1e608b5d856bca7c91f4df0450c35df44d3dcf run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: task hung in new_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in new_device_store run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: task hung in rtnl_newlink run #6: crashed: INFO: rcu detected stall in corrupted run #7: crashed: INFO: task hung in addrconf_dad_work run #8: crashed: INFO: rcu detected stall in corrupted run #9: crashed: INFO: task hung in rtnl_newlink representative crash: INFO: rcu detected stall in corrupted, types: [HANG] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [ATOMIC_SLEEP LEAK UBSAN BUG KASAN LOCKDEP], they are not needed testing commit cfba9f07a1d6aeca38f47f1f472cfb0ba133d341 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 7a069f388241885dbcf9e2ab0f386330baf8ac77522f6b21e6e859f5813752f6 all runs: OK false negative chance: 0.000 minimized to 1273 configs; suspects: [6LOWPAN ARCH_ENABLE_MEMORY_HOTREMOVE ASUS_WMI ATM AX25 BCMA BLK_DEV_ZONED BPF_SYSCALL BRIDGE BRIDGE_NETFILTER CAN CARDBUS CFG80211 CFG80211_WEXT CHARGER_BQ24190 CHECKPOINT_RESTORE CMA COMMON_CLK CONTIG_ALLOC DAX DLM DRM DRM_BRIDGE DRM_I915 DRM_I915_USERPTR DRM_KMS_HELPER DRM_MIPI_DSI DRM_PANEL DRM_PANEL_BRIDGE DRM_PANEL_EDP DRM_PANEL_ORIENTATION_QUIRKS DRM_SIMPLEDRM DRM_TTM DRM_TTM_HELPER DRM_UDL DRM_VGEM DRM_VIRTIO_GPU DRM_VIRTIO_GPU_KMS DRM_VKMS DRM_VMWGFX DUMMY DVB_AF9013 DVB_AF9033 DVB_AS102 DVB_AS102_FE DVB_B2C2_FLEXCOP DVB_B2C2_FLEXCOP_USB DVB_CORE DVB_DIB3000MB DVB_DIB3000MC DVB_EC100 DVB_GP8PSK_FE DVB_RTL2830 DVB_RTL2832 DVB_RTL2832_SDR DVB_TEST_DRIVERS DVB_TTUSB_BUDGET DVB_TTUSB_DEC DVB_USB DVB_USB_A800 DVB_USB_AF9005 DVB_USB_AF9005_REMOTE DVB_USB_AF9015 DVB_USB_AF9035 DVB_USB_ANYSEE DVB_USB_AU6610 DVB_USB_AZ6007 DVB_USB_AZ6027 DVB_USB_CE6230 DVB_USB_CINERGY_T2 DVB_USB_CXUSB DVB_USB_CXUSB_ANALOG DVB_USB_DIB0700 DVB_USB_DIB3000MC DVB_USB_DIBUSB_MB DVB_USB_DIBUSB_MC DVB_USB_DIGITV DVB_USB_DTT200U DVB_USB_DTV5100 DVB_USB_DVBSKY DVB_USB_DW2102 DVB_USB_EC168 DVB_USB_GL861 DVB_USB_GP8PSK DVB_USB_LME2510 DVB_USB_M920X DVB_USB_MXL111SF DVB_USB_NOVA_T_USB2 DVB_USB_OPERA1 DVB_USB_PCTV452E DVB_USB_RTL28XXU DVB_USB_TECHNISAT_USB2 DVB_USB_TTUSB2 DVB_USB_UMT_010 DVB_USB_V2 DVB_USB_VP702X DVB_USB_VP7045 DVB_USB_ZD1301 DVB_VIDTV DVB_ZL10353 ECRYPT_FS ECRYPT_FS_MESSAGING EDAC EEPROM_93CX6 EFS_FS ENCRYPTED_KEYS EQUALIZER EROFS_FS EROFS_FS_BACKED_BY_FILE EROFS_FS_POSIX_ACL EROFS_FS_SECURITY EROFS_FS_XATTR EROFS_FS_ZIP EVM EVM_ADD_XATTRS EVM_ATTR_FSUUID EXFAT_FS EXPORTFS_BLOCK_OPS EXT3_FS EXT3_FS_POSIX_ACL EXT3_FS_SECURITY EXTCON EXTCON_INTEL_CHT_WC EXTCON_PTN5150 EXTCON_USBC_TUSB320 F2FS_CHECK_FS F2FS_FAULT_INJECTION F2FS_FS F2FS_FS_COMPRESSION F2FS_FS_LZ4 F2FS_FS_LZ4HC F2FS_FS_LZO F2FS_FS_LZORLE F2FS_FS_POSIX_ACL F2FS_FS_SECURITY F2FS_FS_XATTR F2FS_FS_ZSTD F2FS_STAT_FS FANOTIFY FANOTIFY_ACCESS_PERMISSIONS FB FB_CFB_COPYAREA FB_CFB_FILLRECT FB_CFB_IMAGEBLIT FB_CORE FB_DEFERRED_IO FB_DEVICE FB_IOMEM_FOPS FB_IOMEM_HELPERS FB_NOTIFY FB_SYSMEM_FOPS FB_SYSMEM_HELPERS FB_SYSMEM_HELPERS_DEFERRED FB_SYS_COPYAREA FB_SYS_FILLRECT FB_SYS_IMAGEBLIT FB_TILEBLITTING FB_VESA FB_VGA16 FB_VIRTUAL FDDI FIREWIRE FIREWIRE_NET FIREWIRE_OHCI FIREWIRE_SBP2 FONT_8x16 FONT_8x8 FONT_SUPPORT FRAMEBUFFER_CONSOLE FRAMEBUFFER_CONSOLE_DETECT_PRIMARY FRAMEBUFFER_CONSOLE_ROTATION FS_DAX FS_DAX_PMD FS_ENCRYPTION FS_ENCRYPTION_ALGS FS_STACK FS_VERITY FS_VERITY_BUILTIN_SIGNATURES FTL FUSE_DAX FUSE_FS FW_LOADER_COMPRESS FW_LOADER_PAGED_BUF FW_LOADER_SYSFS FW_LOADER_USER_HELPER FW_LOADER_USER_HELPER_FALLBACK GACT_PROB GARP GENERIC_PHY GET_FREE_REGION GFS2_FS GFS2_FS_LOCKING_DLM GNSS GNSS_USB GOOGLE_COREBOOT_TABLE GOOGLE_FIRMWARE GOOGLE_MEMCONSOLE GOOGLE_MEMCONSOLE_COREBOOT GOOGLE_VPD GPIOLIB GPIOLIB_IRQCHIP GPIO_ACPI GPIO_DLN2 GPIO_LJCA GPIO_VIPERBOARD GREENASIA_FF GREYBUS GREYBUS_BRIDGED_PHY GREYBUS_ES2 GREYBUS_HID GREYBUS_USB GROUP_SCHED_WEIGHT GTP GUEST_PERF_EVENTS GVE HAMRADIO HAVE_ARCH_NODE_DEV_GROUP HAVE_ARCH_USERFAULTFD_MINOR HAVE_ARCH_USERFAULTFD_WP HAVE_BOOTMEM_INFO_NODE HAVE_CLK_PREPARE HAVE_KVM_CPU_RELAX_INTERCEPT HAVE_KVM_DIRTY_RING HAVE_KVM_DIRTY_RING_ACQ_REL HAVE_KVM_DIRTY_RING_TSO HAVE_KVM_IRQCHIP HAVE_KVM_IRQ_BYPASS HAVE_KVM_IRQ_ROUTING HAVE_KVM_MSI HAVE_KVM_NO_POLL HAVE_KVM_PFNCACHE HAVE_KVM_PM_NOTIFIER HAVE_KVM_READONLY_MEM HAVE_SCHED_AVG_IRQ HDLC HDLC_CISCO HDLC_FR HDLC_PPP HDLC_RAW HDLC_RAW_ETH HDLC_X25 HDMI HFSPLUS_FS HFS_FS HID_ACCUTOUCH HID_ACRUX HID_ACRUX_FF HID_ALPS HID_APPLEIR HID_ASUS HID_AUREAL HID_BATTERY_STRENGTH HID_BETOP_FF HID_BIGBEN_FF HID_CMEDIA HID_CORSAIR HID_COUGAR HID_CP2112 HID_CREATIVE_SB0540 HID_ELAN HID_ELECOM HID_ELO HID_EMS_FF HID_EVISION HID_FT260 HID_GEMBIRD HID_GFRM HID_GLORIOUS HID_GOOGLE_STADIA_FF HID_GREENASIA HID_GT683R HID_HOLTEK HID_ICADE HID_JABRA HID_KEYTOUCH HID_KYE HID_LCPOWER HID_LED HID_LENOVO HID_LETSKETCH HID_LOGITECH_DJ HID_LOGITECH_HIDPP HID_MACALLY HID_MAGICMOUSE HID_MALTRON HID_MAYFLASH HID_MCP2200 HID_MCP2221 HID_MEGAWORLD_FF HID_MULTITOUCH HID_NINTENDO HID_NTI HID_ORTEK HID_PENMOUNT HID_PICOLCD HID_PICOLCD_BACKLIGHT HID_PICOLCD_CIR HID_PICOLCD_FB HID_PICOLCD_LCD HID_PICOLCD_LEDS HID_PLANTRONICS HID_PRIMAX HID_PRODIKEYS HID_PXRC HID_RAZER HID_RETRODE HID_RMI HID_ROCCAT HID_SAITEK HID_SEMITEK HID_SENSOR_ACCEL_3D HID_SENSOR_ALS HID_SENSOR_CUSTOM_INTEL_HINGE HID_SENSOR_CUSTOM_SENSOR HID_SENSOR_DEVICE_ROTATION HID_SENSOR_GYRO_3D HID_SENSOR_HUB HID_SENSOR_HUMIDITY HID_SENSOR_IIO_COMMON HID_SENSOR_IIO_TRIGGER HID_SENSOR_INCLINOMETER_3D HID_SENSOR_MAGNETOMETER_3D HID_SENSOR_PRESS HID_SENSOR_PROX HID_SENSOR_TEMP HID_SIGMAMICRO HID_SPEEDLINK HID_STEELSERIES HID_THINGM HID_TIVO HID_TOPRE HID_TWINHAN HID_U2FZERO HID_UCLOGIC HID_UDRAW_PS3 HID_VIEWSONIC HID_VIVALDI HID_VIVALDI_COMMON HID_VRC2 HID_WACOM HID_WALTOP HID_WIIMOTE HID_XIAOMI HID_XINMO HID_ZEROPLUS HID_ZYDACRON HMM_MIRROR HOLTEK_FF HOTPLUG_PCI_PCIE HPET_MMAP HPET_MMAP_DEFAULT HPFS_FS HSR I2C_ALGOBIT I2C_CHARDEV I2C_CP2615 I2C_DESIGNWARE_CORE I2C_DESIGNWARE_PLATFORM I2C_DIOLAN_U2C I2C_DLN2 I2C_HID_ACPI I2C_HID_CORE I2C_HID_OF I2C_LJCA I2C_MUX I2C_MUX_REG I2C_ROBOTFUZZ_OSIF I2C_SI4713 I2C_SLAVE I2C_SLAVE_EEPROM I2C_TINY_USB I2C_VIPERBOARD IEEE802154 IEEE802154_6LOWPAN IEEE802154_ATUSB IEEE802154_DRIVERS IEEE802154_HWSIM IEEE802154_NL802154_EXPERIMENTAL IEEE802154_SOCKET IFB IIO IIO_BUFFER IIO_KFIFO_BUF IIO_TRIGGER IIO_TRIGGERED_BUFFER IKCONFIG IKCONFIG_PROC IMA IMA_APPRAISE IMA_APPRAISE_MODSIG IMA_DEFAULT_HASH_SHA256 IMA_LSM_RULES IMA_MEASURE_ASYMMETRIC_KEYS IMA_NG_TEMPLATE IMA_QUEUE_EARLY_BOOT_KEYS IMA_READ_POLICY IMA_WRITE_POLICY INET6_ESPINTCP INET6_ESP_OFFLOAD INET6_IPCOMP INET6_TUNNEL INET6_XFRM_TUNNEL INET_AH INET_DCCP_DIAG INET_DIAG INET_DIAG_DESTROY INET_ESP INET_ESPINTCP INET_ESP_OFFLOAD INET_IPCOMP INET_MPTCP_DIAG INET_RAW_DIAG INET_SCTP_DIAG INET_TCP_DIAG INET_UDP_DIAG INET_XFRM_TUNNEL INFINIBAND INFINIBAND_ADDR_TRANS INFINIBAND_ADDR_TRANS_CONFIGFS INFINIBAND_IPOIB INFINIBAND_IPOIB_CM INFINIBAND_IPOIB_DEBUG INFINIBAND_ISER INFINIBAND_ON_DEMAND_PAGING INFINIBAND_RTRS INFINIBAND_SRP INFINIBAND_USER_ACCESS INFINIBAND_USER_MAD INFINIBAND_USER_MEM INPUT_ATI_REMOTE2 INPUT_CM109 INPUT_IMS_PCU INPUT_JOYDEV INPUT_JOYSTICK INPUT_KEYSPAN_REMOTE INPUT_LEDS INPUT_MOUSE INPUT_MOUSEDEV INPUT_MOUSEDEV_PSAUX INPUT_POWERMATE INPUT_UINPUT INPUT_YEALINK INTEGRITY INTEGRITY_ASYMMETRIC_KEYS INTEGRITY_AUDIT INTEGRITY_SIGNATURE INTEGRITY_TRUSTED_KEYRING INTEL_CHTWC_INT33FE INTEL_IDMA64 INTEL_IOATDMA INTEL_IOMMU_DEFAULT_ON INTEL_IOMMU_SVM INTEL_ISHTP_ECLITE INTEL_ISH_FIRMWARE_DOWNLOADER INTEL_ISH_HID INTEL_SOC_PMIC_CHTWC INTERVAL_TREE_SPAN_ITER IOMMUFD IOMMUFD_DRIVER IOMMUFD_DRIVER_CORE IOMMUFD_TEST IP6_NF_MATCH_AH IP6_NF_MATCH_EUI64 IP6_NF_MATCH_FRAG IP6_NF_MATCH_HL IP6_NF_MATCH_MH IP6_NF_MATCH_OPTS IP6_NF_MATCH_RPFILTER IP6_NF_MATCH_RT IP6_NF_MATCH_SRH IP6_NF_NAT IP6_NF_RAW IP6_NF_SECURITY IP6_NF_TARGET_HL IP6_NF_TARGET_MASQUERADE IP6_NF_TARGET_NPT IP6_NF_TARGET_SYNPROXY IPV6_FOU IPV6_FOU_TUNNEL IPV6_GRE IPV6_ILA IPV6_MIP6 IPV6_MROUTE IPV6_MROUTE_MULTIPLE_TABLES IPV6_MULTIPLE_TABLES IPV6_OPTIMISTIC_DAD IPV6_PIMSM_V2 IPV6_ROUTER_PREF IPV6_ROUTE_INFO IPV6_RPL_LWTUNNEL IPV6_SEG6_BPF IPV6_SEG6_HMAC IPV6_SEG6_LWTUNNEL IPV6_SIT_6RD IPV6_SUBTREES IPV6_TUNNEL IPV6_VTI IPVLAN IPVLAN_L3S IPVTAP IP_DCCP IP_DCCP_CCID3 IP_DCCP_TFRC_LIB IP_FIB_TRIE_STATS IP_MROUTE_MULTIPLE_TABLES IP_NF_ARPFILTER IP_NF_ARPTABLES IP_NF_ARP_MANGLE IP_NF_MATCH_AH IP_NF_MATCH_ECN IP_NF_MATCH_RPFILTER IP_NF_MATCH_TTL IP_NF_RAW IP_NF_SECURITY IP_NF_TARGET_ECN IP_NF_TARGET_NETMAP IP_NF_TARGET_REDIRECT IP_NF_TARGET_SYNPROXY IP_NF_TARGET_TTL IP_ROUTE_CLASSID IP_SCTP IP_SET IP_SET_BITMAP_IP IP_SET_BITMAP_IPMAC IP_SET_BITMAP_PORT IP_SET_HASH_IP IP_SET_HASH_IPMAC IP_SET_HASH_IPMARK IP_SET_HASH_IPPORT IP_SET_HASH_IPPORTIP IP_SET_HASH_IPPORTNET IP_SET_HASH_MAC IP_SET_HASH_NET IP_SET_HASH_NETIFACE IP_SET_HASH_NETNET IP_SET_HASH_NETPORT IP_SET_HASH_NETPORTNET IP_SET_LIST_SET IP_VS IP_VS_DH IP_VS_FO IP_VS_FTP IP_VS_IPV6 IP_VS_LBLC IP_VS_LBLCR IP_VS_LC IP_VS_MH IP_VS_NFCT IP_VS_NQ IP_VS_OVF IP_VS_PE_SIP IP_VS_PROTO_AH IP_VS_PROTO_AH_ESP IP_VS_PROTO_ESP IP_VS_PROTO_SCTP IP_VS_PROTO_TCP IP_VS_PROTO_UDP IP_VS_RR IP_VS_SED IP_VS_SH IP_VS_TWOS IP_VS_WLC IP_VS_WRR IRQ_BYPASS_MANAGER IRQ_POLL IRQ_REMAP IRQ_TIME_ACCOUNTING IR_IGORPLUGUSB IR_IGUANA IR_IMON IR_IMON_RAW IR_MCEUSB IR_REDRAT3 IR_STREAMZAP IR_TOY IR_TTUSBIR ISDN ISDN_CAPI JFFS2_CMODE_PRIORITY JFFS2_COMPRESSION_OPTIONS JFFS2_FS JFFS2_FS_POSIX_ACL JFFS2_FS_SECURITY JFFS2_FS_WRITEBUFFER JFFS2_FS_XATTR JFFS2_LZO JFFS2_RTIME JFFS2_RUBIN JFFS2_SUMMARY JFFS2_ZLIB JFS_DEBUG JFS_FS JFS_POSIX_ACL JFS_SECURITY JOYSTICK_IFORCE JOYSTICK_IFORCE_USB JOYSTICK_PXRC JOYSTICK_XPAD JOYSTICK_XPAD_FF JOYSTICK_XPAD_LEDS KARMA_PARTITION KCOV KCOV_ENABLE_COMPARISONS KCOV_INSTRUMENT_ALL KEYS_REQUEST_CACHE KEY_DH_OPERATIONS KEY_NOTIFICATIONS KSM KVM KVM_AMD KVM_ASYNC_PF KVM_COMMON KVM_COMPAT KVM_ELIDE_TLB_FLUSH_IF_YOUNG KVM_GENERIC_DIRTYLOG_READ_PROTECT KVM_GENERIC_HARDWARE_ENABLING KVM_GENERIC_MEMORY_ATTRIBUTES KVM_GENERIC_MMU_NOTIFIER KVM_GENERIC_PRE_FAULT_MEMORY KVM_GENERIC_PRIVATE_MEM KVM_HYPERV KVM_INTEL KVM_MMIO KVM_PRIVATE_MEM KVM_PROVE_MMU KVM_SW_PROTECTED_VM KVM_VFIO KVM_X86 KVM_XEN KVM_XFER_TO_GUEST_WORK L2TP L2TP_ETH L2TP_IP L2TP_V3 LAPB LAPBETHER LCD_CLASS_DEVICE LDM_PARTITION LEGACY_PTYS LIBCRC32C LIBNVDIMM LINEAR_RANGES LLC LLC2 LOGIG940_FF LOGIRUMBLEPAD2_FF LOGO LOGO_LINUX_MONO LOGO_LINUX_VGA16 LPC_ICH LWTUNNEL LWTUNNEL_BPF LZ4HC_COMPRESS LZ4_COMPRESS MAC80211 MAC80211_HAS_RC MAC80211_HWSIM MAC80211_MESH MAC80211_RC_DEFAULT_MINSTREL MAC80211_RC_MINSTREL MAC802154 MACSEC MACVLAN MACVTAP MAC_PARTITION MAPPING_DIRTY_HELPERS MDIO_MVUSB MD_RAID0 MD_RAID1 MD_RAID10 MD_RAID456 MEDIA_ANALOG_TV_SUPPORT MEDIA_ATTACH MEDIA_CAMERA_SUPPORT MEDIA_CEC_SUPPORT MEDIA_CONTROLLER MEDIA_CONTROLLER_DVB MEDIA_DIGITAL_TV_SUPPORT MEDIA_RADIO_SUPPORT MEDIA_SDR_SUPPORT MEDIA_SUPPORT MEDIA_SUPPORT_FILTER MEDIA_TEST_SUPPORT MEDIA_TUNER MEDIA_TUNER_MSI001 MEDIA_USB_SUPPORT MEMORY_BALLOON MEMORY_HOTPLUG MEMORY_HOTPLUG_DEFAULT_ONLINE MEMORY_HOTREMOVE MEMORY_ISOLATION MEMREGION MEMSTICK MEMSTICK_REALTEK_USB MEM_SOFT_DIRTY MFD_CORE MFD_DLN2 MFD_SYSCON MFD_VIPERBOARD MHI_BUS MHI_WWAN_CTRL MHP_MEMMAP_ON_MEMORY MICROCHIP_PHY MINIX_FS MINIX_SUBPARTITION MIN_HEAP MISC_RTSX MISC_RTSX_USB MISDN MISDN_DSP MISDN_HFCUSB MISDN_L1OIP MKISS MLX4_CORE MLX4_INFINIBAND MMC MMC_REALTEK_USB MMC_USHC MMC_VUB300 MODULE_SRCVERSION_ALL MODVERSIONS MOST MOST_USB_HDM MOUSE_APPLETOUCH MOUSE_BCM5974 MOUSE_PS2 MOUSE_PS2_ALPS MOUSE_PS2_BYD MOUSE_PS2_CYPRESS MOUSE_PS2_FOCALTECH MOUSE_PS2_LIFEBOOK MOUSE_PS2_LOGIPS2PP MOUSE_PS2_SMBUS MOUSE_PS2_SYNAPTICS MOUSE_PS2_SYNAPTICS_SMBUS MOUSE_PS2_TRACKPOINT MOUSE_SYNAPTICS_USB MPLS MPLS_IPTUNNEL MPLS_ROUTING MPTCP MPTCP_IPV6 MRP MTD MTD_BLKDEVS MTD_BLOCK MTD_BLOCK2MTD MTD_CFI_I1 MTD_CFI_I2 MTD_MAP_BANK_WIDTH_1 MTD_MAP_BANK_WIDTH_2 MTD_MAP_BANK_WIDTH_4 MTD_MTDRAM MTD_PHRAM MTD_SLRAM MUSB_PIO_ONLY ND_BTT ND_CLAIM ND_PFN NETDEVSIM NETFILTER_ADVANCED NETFILTER_BPF_LINK NETFILTER_FAMILY_ARP NETFILTER_FAMILY_BRIDGE NETFILTER_NETLINK_ACCT NETFILTER_NETLINK_GLUE_CT NETFILTER_NETLINK_OSF NETFILTER_NETLINK_QUEUE NETFILTER_SYNPROXY NETFILTER_XTABLES_COMPAT NETFILTER_XT_CONNMARK NETFILTER_XT_MATCH_BPF NETFILTER_XT_MATCH_CGROUP NETFILTER_XT_MATCH_CLUSTER NETFILTER_XT_MATCH_COMMENT NETFILTER_XT_MATCH_CONNBYTES NETFILTER_XT_MATCH_CONNLABEL NETFILTER_XT_MATCH_CONNLIMIT NETFILTER_XT_MATCH_CONNMARK NETFILTER_XT_MATCH_CPU NETFILTER_XT_MATCH_DCCP NETFILTER_XT_MATCH_DEVGROUP NETFILTER_XT_MATCH_DSCP NETFILTER_XT_MATCH_ECN NETFILTER_XT_MATCH_ESP NETFILTER_XT_MATCH_HASHLIMIT NETFILTER_XT_MATCH_HELPER NETFILTER_XT_MATCH_HL NETFILTER_XT_MATCH_IPCOMP NETFILTER_XT_MATCH_IPRANGE NETFILTER_XT_MATCH_IPVS NETFILTER_XT_MATCH_L2TP NETFILTER_XT_MATCH_LENGTH NETFILTER_XT_MATCH_LIMIT NETFILTER_XT_MATCH_MAC NETFILTER_XT_MATCH_MARK NETFILTER_XT_MATCH_MULTIPORT NETFILTER_XT_MATCH_NFACCT NETFILTER_XT_MATCH_OSF NETFILTER_XT_MATCH_OWNER NETFILTER_XT_MATCH_PHYSDEV NETFILTER_XT_MATCH_PKTTYPE NETFILTER_XT_MATCH_QUOTA NETFILTER_XT_MATCH_RATEEST NETFILTER_XT_MATCH_REALM NETFILTER_XT_MATCH_RECENT NETFILTER_XT_MATCH_SCTP NETFILTER_XT_MATCH_SOCKET NETFILTER_XT_MATCH_STATISTIC NETFILTER_XT_MATCH_STRING NETFILTER_XT_MATCH_TCPMSS NETFILTER_XT_MATCH_TIME NETFILTER_XT_MATCH_U32 NETFILTER_XT_SET NETFILTER_XT_TARGET_AUDIT NETFILTER_XT_TARGET_CHECKSUM NETFILTER_XT_TARGET_CLASSIFY NETFILTER_XT_TARGET_CONNMARK NETFILTER_XT_TARGET_CT NETFILTER_XT_TARGET_DSCP NETFILTER_XT_TARGET_HL NETFILTER_XT_TARGET_HMARK NETFILTER_XT_TARGET_IDLETIMER NETFILTER_XT_TARGET_LED NETFILTER_XT_TARGET_MARK NETFILTER_XT_TARGET_NETMAP NETFILTER_XT_TARGET_NFQUEUE NETFILTER_XT_TARGET_NOTRACK NETFILTER_XT_TARGET_RATEEST NETFILTER_XT_TARGET_REDIRECT NETFILTER_XT_TARGET_TCPOPTSTRIP NETFILTER_XT_TARGET_TEE NETFILTER_XT_TARGET_TPROXY NETFILTER_XT_TARGET_TRACE NETLABEL NETLINK_DIAG NETROM NET_9P_RDMA NET_ACT_BPF NET_ACT_CONNMARK NET_ACT_CSUM NET_ACT_CT NET_ACT_CTINFO NET_ACT_GACT NET_ACT_GATE NET_ACT_IFE NET_ACT_MIRRED NET_ACT_MPLS NET_ACT_NAT NET_ACT_PEDIT NET_ACT_POLICE NET_ACT_SAMPLE NET_ACT_SIMP NET_ACT_SKBEDIT NET_ACT_SKBMOD NET_ACT_TUNNEL_KEY NET_ACT_VLAN NET_CLS_BASIC NET_CLS_BPF NET_CLS_FLOW NET_CLS_FLOWER NET_CLS_FW NET_CLS_MATCHALL NET_CLS_ROUTE4 NET_DEVLINK NET_DEVMEM NET_DROP_MONITOR NET_DSA NET_DSA_TAG_BRCM NET_DSA_TAG_BRCM_COMMON NET_DSA_TAG_BRCM_PREPEND NET_DSA_TAG_MTK NET_DSA_TAG_QCA NET_DSA_TAG_RTL4_A NET_EMATCH_CANID NET_EMATCH_CMP NET_EMATCH_IPSET NET_EMATCH_IPT NET_EMATCH_META NET_EMATCH_NBYTE NET_EMATCH_TEXT NET_EMATCH_U32 NET_FC NET_FOU NET_FOU_IP_TUNNELS NET_IFE NET_IFE_SKBMARK NET_IFE_SKBPRIO NET_IFE_SKBTCINDEX NET_IPGRE NET_IPGRE_BROADCAST NET_IPGRE_DEMUX NET_IPIP NET_IPVTI NET_KEY NET_KEY_MIGRATE NET_L3_MASTER_DEV NET_MPLS_GSO NET_NCSI NET_NSH NET_REDIRECT NET_SCH_CAKE NET_SCH_CBS NET_SCH_CHOKE NET_SCH_CODEL NET_SCH_DRR NET_SCH_ETF NET_SCH_ETS NET_SCH_FQ NET_SCH_FQ_CODEL NET_SCH_FQ_PIE NET_SCH_GRED NET_SCH_HFSC NET_SCH_HHF NET_SCH_HTB NET_SCH_INGRESS NET_SCH_MQPRIO NET_SCH_MQPRIO_LIB NET_SCH_MULTIQ NET_SCH_NETEM NET_SCH_PIE NET_SCH_PLUG NET_SCH_PRIO NET_SCH_QFQ NET_SCH_RED NET_SCH_SFB NET_SCH_SFQ NET_SCH_SKBPRIO NET_SCH_TAPRIO NET_SCH_TBF NET_SCH_TEQL NET_SHAPER NET_SOCK_MSG NET_SWITCHDEV NET_TC_SKB_EXT NET_TEAM NET_TEAM_MODE_ACTIVEBACKUP NET_TEAM_MODE_BROADCAST NET_TEAM_MODE_LOADBALANCE NET_TEAM_MODE_RANDOM NET_TEAM_MODE_ROUNDROBIN NET_UDP_TUNNEL NET_VENDOR_HISILICON NET_VRF NFC NFC_DIGITAL NFC_FDP NFC_HCI NFC_MRVL NFC_MRVL_USB NFC_NCI NFC_NCI_UART NFC_PN533 NFC_PN533_USB NFC_PORT100 NFC_SHDLC NFC_SIM NFC_VIRTUAL_NCI NFSD NFSD_BLOCKLAYOUT NFSD_FLEXFILELAYOUT NFSD_PNFS NFSD_SCSILAYOUT NFSD_V3_ACL NFSD_V4 NFSD_V4_2_INTER_SSC NFSD_V4_SECURITY_LABEL NFS_FSCACHE NFS_V4_1 NFS_V4_2 NFS_V4_2_READ_PLUS NFS_V4_2_SSC_HELPER NFS_V4_SECURITY_LABEL NFT_BRIDGE_META NFT_BRIDGE_REJECT NFT_COMPAT NFT_COMPAT_ARP NFT_CONNLIMIT NFT_CT NFT_DUP_IPV4 NFT_DUP_IPV6 NFT_DUP_NETDEV NFT_FIB NFT_FIB_INET NFT_FIB_IPV4 NFT_FIB_IPV6 NFT_FIB_NETDEV NFT_FLOW_OFFLOAD NFT_FWD_NETDEV NFT_HASH NFT_LIMIT NFT_LOG NFT_MASQ NFT_NAT NFT_NUMGEN NFT_OSF NFT_QUEUE NFT_QUOTA NFT_REDIR NFT_REJECT NFT_REJECT_INET NFT_REJECT_IPV4 NFT_REJECT_IPV6 NFT_REJECT_NETDEV NFT_SOCKET NFT_SYNPROXY NFT_TPROXY NFT_TUNNEL NFT_XFRM NF_CONNTRACK_AMANDA NF_CONNTRACK_BRIDGE NF_CONNTRACK_BROADCAST NF_CONNTRACK_EVENTS NF_CONNTRACK_H323 NF_CONNTRACK_LABELS NF_CONNTRACK_MARK NF_CONNTRACK_NETBIOS_NS NF_CONNTRACK_OVS NF_CONNTRACK_PPTP NF_CONNTRACK_SANE NF_CONNTRACK_SNMP NF_CONNTRACK_TFTP NF_CONNTRACK_TIMEOUT NF_CONNTRACK_TIMESTAMP NF_CONNTRACK_ZONES NF_CT_NETLINK_HELPER NF_CT_NETLINK_TIMEOUT NF_CT_PROTO_DCCP NF_CT_PROTO_GRE NF_CT_PROTO_SCTP NF_CT_PROTO_UDPLITE NF_DUP_IPV4 NF_DUP_IPV6 NF_DUP_NETDEV NF_FLOW_TABLE NF_FLOW_TABLE_INET NF_NAT_AMANDA NF_NAT_H323 NF_NAT_OVS NF_NAT_PPTP NF_NAT_REDIRECT NF_NAT_SNMP_BASIC NF_NAT_TFTP NF_SOCKET_IPV4 NF_SOCKET_IPV6 NF_TABLES NF_TABLES_ARP NF_TABLES_BRIDGE NF_TABLES_INET NF_TABLES_IPV4 NF_TABLES_IPV6 NF_TABLES_NETDEV NF_TPROXY_IPV4 NF_TPROXY_IPV6 NILFS2_FS NINTENDO_FF NLMON NLS_CODEPAGE_1250 NLS_CODEPAGE_1251 NLS_CODEPAGE_737 NOP_USB_XCEIV PAGE_POOL PARPORT PARTITION_ADVANCED PCCARD PCMCIA PHONET PSAMPLE RADIO_ADAPTERS RADIO_SI470X RADIO_SI4713 RAS RC_CORE RC_DEVICES REGULATOR RFKILL SND SOUND SPI SSB STAGING TAP TARGET_CORE TRANSPARENT_HUGEPAGE TUN TYPEC TYPEC_MUX_PI3USB30532 USB_AMD5536UDC USB_ATM USB_CDNS3 USB_CDNS3_PCI_WRAP USB_CDNSP_GADGET USB_CDNSP_HOST USB_CDNSP_PCI USB_CDNS_HOST USB_CDNS_SUPPORT USB_CHAOSKEY USB_CHIPIDEA USB_CHIPIDEA_GENERIC USB_CHIPIDEA_HOST USB_CHIPIDEA_MSM USB_CHIPIDEA_NPCM USB_CHIPIDEA_PCI USB_CHIPIDEA_UDC USB_CONFIGFS USB_CONFIGFS_ACM USB_CONFIGFS_ECM USB_CONFIGFS_ECM_SUBSET USB_CONFIGFS_EEM USB_CONFIGFS_F_FS USB_CONFIGFS_F_HID USB_CONFIGFS_F_LB_SS USB_CONFIGFS_F_MIDI USB_CONFIGFS_F_MIDI2 USB_CONFIGFS_F_PRINTER USB_CONFIGFS_F_TCM USB_CONFIGFS_F_UAC1 USB_CONFIGFS_F_UAC1_LEGACY USB_CONFIGFS_F_UAC2 USB_CONFIGFS_F_UVC USB_CONFIGFS_MASS_STORAGE USB_CONFIGFS_NCM USB_CONFIGFS_OBEX USB_CONFIGFS_PHONET USB_CONFIGFS_RNDIS USB_CONFIGFS_SERIAL USB_CONN_GPIO USB_CXACRU USB_CYPRESS_CY7C63 USB_CYTHERM USB_DSBR USB_DUMMY_HCD USB_DWC2 USB_DWC2_HOST USB_DWC2_PCI USB_DWC3 USB_DWC3_GADGET USB_DWC3_HAPS USB_DWC3_OF_SIMPLE USB_DWC3_PCI USB_DWC3_ULPI USB_DYNAMIC_MINORS USB_EG20T USB_EHCI_FSL USB_EHCI_HCD_PLATFORM USB_EHCI_ROOT_HUB_TT USB_EHSET_TEST_FIXTURE USB_EMI26 USB_EMI62 USB_EPSON2888 USB_EZUSB_FX2 USB_FEW_INIT_RETRIES USB_F_ACM USB_F_ECM USB_F_EEM USB_F_FS USB_F_HID USB_F_MASS_STORAGE USB_F_MIDI USB_F_MIDI2 USB_F_NCM USB_F_OBEX USB_F_PHONET USB_F_PRINTER USB_F_RNDIS USB_F_SERIAL USB_F_SS_LB USB_F_SUBSET USB_F_TCM USB_F_UAC1 USB_F_UAC1_LEGACY USB_F_UAC2 USB_F_UVC USB_GADGET USB_GADGETFS USB_GADGET_DEBUG_FILES USB_GADGET_DEBUG_FS USB_GL860 USB_GOKU USB_GR_UDC USB_GSPCA USB_GSPCA_BENQ USB_GSPCA_CONEX USB_GSPCA_CPIA1 USB_GSPCA_DTCS033 USB_GSPCA_ETOMS USB_GSPCA_FINEPIX USB_GSPCA_JEILINJ USB_GSPCA_JL2005BCD USB_GSPCA_KINECT USB_GSPCA_KONICA USB_GSPCA_MARS USB_GSPCA_MR97310A USB_GSPCA_NW80X USB_GSPCA_OV519 USB_GSPCA_OV534 USB_GSPCA_OV534_9 USB_GSPCA_PAC207 USB_GSPCA_PAC7302 USB_GSPCA_PAC7311 USB_GSPCA_SE401 USB_GSPCA_SN9C2028 USB_GSPCA_SN9C20X USB_GSPCA_SONIXB USB_GSPCA_SONIXJ USB_GSPCA_SPCA1528 USB_GSPCA_SPCA500 USB_GSPCA_SPCA501 USB_GSPCA_SPCA505 USB_GSPCA_SPCA506 USB_GSPCA_SPCA508 USB_GSPCA_SPCA561 USB_GSPCA_SQ905 USB_GSPCA_SQ905C USB_GSPCA_SQ930X USB_GSPCA_STK014 USB_GSPCA_STK1135 USB_GSPCA_STV0680 USB_GSPCA_SUNPLUS USB_GSPCA_T613 USB_GSPCA_TOPRO USB_GSPCA_TOUPTEK USB_GSPCA_TV8532 USB_GSPCA_VC032X USB_GSPCA_VICAM USB_GSPCA_XIRLINK_CIT USB_GSPCA_ZC3XX USB_HACKRF USB_HCD_BCMA USB_HCD_SSB USB_HSIC_USB3503 USB_HSIC_USB4604 USB_HSO USB_HUB_USB251XB USB_IDMOUSE USB_IOWARRIOR USB_IPHETH USB_ISIGHTFW USB_ISP116X_HCD USB_ISP1301 USB_ISP1760 USB_ISP1760_DUAL_ROLE USB_ISP1760_HCD USB_ISP1761_UDC USB_KAWETH USB_KC2190 USB_KEENE USB_LAN78XX USB_LCD USB_LD USB_LEDS_TRIGGER_USBPORT USB_LED_TRIG USB_LEGOTOWER USB_LGM_PHY USB_LIBCOMPOSITE USB_LINK_LAYER_TEST USB_LJCA USB_M5602 USB_MA901 USB_MAX3420_UDC USB_MAX3421_HCD USB_MDC800 USB_MICROTEK USB_MR800 USB_MSI2500 USB_MUSB_DUAL_ROLE USB_MUSB_HDRC USB_MV_U3D USB_MV_UDC USB_NET2272 USB_NET2272_DMA USB_NET2280 USB_NET_AQC111 USB_NET_AX88179_178A USB_NET_AX8817X USB_NET_CDCETHER USB_NET_CDC_EEM USB_NET_CDC_MBIM USB_NET_CDC_NCM USB_NET_CDC_SUBSET USB_NET_CDC_SUBSET_ENABLE USB_NET_CH9200 USB_NET_CX82310_ETH USB_NET_DM9601 USB_NET_GL620A USB_NET_HUAWEI_CDC_NCM USB_NET_INT51X1 USB_NET_KALMIA USB_NET_MCS7830 USB_NET_NET1080 USB_NET_PLUSB USB_NET_QMI_WWAN USB_NET_RNDIS_HOST USB_NET_SMSC75XX USB_NET_SMSC95XX USB_NET_SR9700 USB_NET_SR9800 USB_NET_ZAURUS USB_OHCI_HCD_PLATFORM USB_OTG USB_OTG_FSM USB_OXU210HP_HCD USB_PEGASUS USB_PULSE8_CEC USB_PWC USB_PWC_INPUT_EVDEV USB_PXA27X USB_R8A66597 USB_R8A66597_HCD USB_RAINSHADOW_CEC USB_RAREMONO USB_RAW_GADGET USB_ROLES_INTEL_XHCI USB_ROLE_SWITCH USB_RTL8150 USB_RTL8152 USB_RTL8153_ECM USB_S2255 USB_SERIAL USB_SERIAL_AIRCABLE USB_SERIAL_ARK3116 USB_SERIAL_BELKIN USB_SERIAL_CH341 USB_SERIAL_CONSOLE USB_SERIAL_CP210X USB_SERIAL_CYBERJACK USB_SERIAL_CYPRESS_M8 USB_SERIAL_DEBUG USB_SERIAL_DIGI_ACCELEPORT USB_SERIAL_EDGEPORT USB_SERIAL_EDGEPORT_TI USB_SERIAL_EMPEG USB_SERIAL_F81232 USB_SERIAL_F8153X USB_SERIAL_FTDI_SIO USB_SERIAL_GARMIN USB_SERIAL_GENERIC USB_SERIAL_IPAQ USB_SERIAL_IPW USB_SERIAL_IR USB_SERIAL_IUU USB_SERIAL_KEYSPAN USB_SERIAL_KEYSPAN_PDA USB_SERIAL_KLSI USB_SERIAL_KOBIL_SCT USB_SERIAL_MCT_U232 USB_SERIAL_METRO USB_SERIAL_MOS7715_PARPORT USB_SERIAL_MOS7720 USB_SERIAL_MOS7840 USB_SERIAL_MXUPORT USB_SERIAL_NAVMAN USB_SERIAL_OMNINET USB_SERIAL_OPTICON USB_SERIAL_OPTION USB_SERIAL_OTI6858 USB_SERIAL_PL2303 USB_SERIAL_QCAUX USB_SERIAL_QT2 USB_SERIAL_QUALCOMM USB_SERIAL_SAFE USB_SERIAL_SIERRAWIRELESS USB_SERIAL_SIMPLE USB_SERIAL_SPCP8X5 USB_SERIAL_SSU100 USB_SERIAL_SYMBOL USB_SERIAL_TI USB_SERIAL_UPD78F0730 USB_SERIAL_VISOR USB_SERIAL_WHITEHEAT USB_SERIAL_WISHBONE USB_SERIAL_WWAN USB_SERIAL_XR USB_SERIAL_XSENS_MT USB_SEVSEG USB_SI470X USB_SI4713 USB_SIERRA_NET USB_SISUSBVGA USB_SL811_CS USB_SL811_HCD USB_SL811_HCD_ISO USB_SNP_CORE USB_SPEEDTOUCH USB_STORAGE_ALAUDA USB_STORAGE_CYPRESS_ATACB USB_STORAGE_DATAFAB USB_STORAGE_ENE_UB6250 USB_STORAGE_FREECOM USB_STORAGE_ISD200 USB_STORAGE_JUMPSHOT USB_STORAGE_KARMA USB_STORAGE_ONETOUCH USB_STORAGE_SDDR09 USB_STORAGE_SDDR55 USB_STORAGE_USBAT USB_STV06XX USB_TEST USB_TMC USB_TRANCEVIBRATOR USB_UAS USB_UEAGLEATM USB_ULPI_BUS USB_USBNET USB_USS720 USB_U_AUDIO USB_U_ETHER USB_U_SERIAL USB_VIDEO_CLASS USB_VIDEO_CLASS_INPUT_EVDEV USB_VL600 USB_WDM USB_XHCI_DBGCAP USB_XHCI_PCI_RENESAS USB_XHCI_PLATFORM USB_XUSBATM USB_YUREX USERFAULTFD USERIO USERMODE_DRIVER USER_RETURN_NOTIFIER UVC_COMMON U_SERIAL_CONSOLE V4L2_MEM2MEM_DEV V4L_TEST_DRIVERS VALIDATE_FS_PARSER VDPA VDPA_SIM VDPA_SIM_BLOCK VDPA_SIM_NET VETH VFIO VFIO_DEVICE_CDEV VFIO_PCI VFIO_PCI_CORE VFIO_PCI_INTX VFIO_PCI_MMAP VFIO_VIRQFD VGASTATE VHOST VHOST_CROSS_ENDIAN_LEGACY VHOST_IOTLB VHOST_NET VHOST_RING VHOST_TASK VHOST_VDPA VHOST_VSOCK VIDEO VIDEOBUF2_CORE VIDEOBUF2_DMA_CONTIG VIDEOBUF2_DMA_SG VIDEOBUF2_MEMOPS VIDEOBUF2_V4L2 VIDEOBUF2_VMALLOC VIDEOMODE_HELPERS VIDEO_AU0828 VIDEO_AU0828_RC VIDEO_AU0828_V4L2 VIDEO_CS53L32A VIDEO_CX231XX VIDEO_CX231XX_ALSA VIDEO_CX231XX_DVB VIDEO_CX231XX_RC VIDEO_CX2341X VIDEO_CX25840 VIDEO_DEV VIDEO_EM28XX VIDEO_EM28XX_ALSA VIDEO_EM28XX_DVB VIDEO_EM28XX_RC VIDEO_EM28XX_V4L2 VIDEO_GO7007 VIDEO_GO7007_LOADER VIDEO_GO7007_USB VIDEO_GO7007_USB_S2250_BOARD VIDEO_HDPVR VIDEO_MSP3400 VIDEO_PVRUSB2 VIDEO_PVRUSB2_DVB VIDEO_PVRUSB2_SYSFS VIDEO_SAA711X VIDEO_STK1160 VIDEO_TUNER VIDEO_TVEEPROM VIDEO_USBTV VIDEO_V4L2_I2C VIDEO_V4L2_SUBDEV_API VIDEO_V4L2_TPG VIDEO_VICODEC VIDEO_VIM2M VIDEO_VIMC VIDEO_VIVID VIDEO_VIVID_CEC VIDEO_WM8775 VIPERBOARD_ADC VIRTIO_BALLOON VIRTIO_DMA_SHARED_BUFFER VIRTIO_FS VIRTIO_MEM VIRTIO_MMIO VIRTIO_MMIO_CMDLINE_DEVICES VIRTIO_PMEM VIRTIO_VDPA VIRTIO_VSOCKETS VIRTIO_VSOCKETS_COMMON VIRT_WIFI VLAN_8021Q VLAN_8021Q_GVRP VLAN_8021Q_MVRP VMAP_PFN VMWARE_VMCI VMXNET3 VP_VDPA VSOCKETS VSOCKETS_DIAG VSOCKETS_LOOPBACK VSOCKMON VT_HW_CONSOLE_BINDING VXFS_FS WAN WANT_DEV_COREDUMP WATCH_QUEUE WEXT_CORE WEXT_PROC WIREGUARD WIRELESS WLAN WLAN_VENDOR_ADMTEK WLAN_VENDOR_SILABS WWAN X25 X86_SGX X86_SGX_KVM X86_USER_SHADOW_STACK X86_X2APIC X86_X32_ABI XDP_SOCKETS XDP_SOCKETS_DIAG XFRM_ESPINTCP XFRM_INTERFACE XFRM_IPCOMP XFRM_MIGRATE XFRM_OFFLOAD XFRM_STATISTICS XFRM_SUB_POLICY XFRM_USER_COMPAT XFS_FS XFS_POSIX_ACL XFS_QUOTA XFS_RT XILLYBUS_CLASS XILLYUSB XOR_BLOCKS YENTA YENTA_ENE_TUNE YENTA_O2 YENTA_RICOH YENTA_TI YENTA_TOSHIBA Z3FOLD Z3FOLD_DEPRECATED ZEROPLUS_FF ZLIB_DEFLATE ZONEFS_FS ZONE_DEVICE ZPOOL ZRAM ZRAM_BACKEND_FORCE_LZO ZRAM_BACKEND_LZO ZRAM_DEF_COMP_LZO ZSMALLOC ZSTD_COMPRESS ZSWAP ZSWAP_COMPRESSOR_DEFAULT_842 ZSWAP_DEFAULT_ON ZSWAP_SHRINKER_DEFAULT_ON ZSWAP_ZPOOL_DEFAULT_Z3FOLD_DEPRECATED] disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed picked [v6.12 v6.11 v6.10 v6.8 v6.6 v6.4 v6.2 v6.0 v5.17 v5.14 v5.11 v5.8 v5.5 v5.2 v4.20 v4.19] out of 35 release tags testing release v6.12 testing commit adc218676eef25575469234709c2d87185ca223a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9b24c4a7ce5cf5707bd65c395a8c0347005ea1a1fd651375e280582b3d36d55b run #0: crashed: INFO: task hung in regdb_fw_cb run #1: crashed: INFO: task hung in new_device_store run #2: crashed: INFO: task hung in new_device_store run #3: crashed: INFO: rcu detected stall in ip_rcv run #4: crashed: INFO: task hung in linkwatch_event run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: task hung in addrconf_dad_work run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: INFO: task hung in addrconf_dad_work run #9: OK representative crash: INFO: task hung in regdb_fw_cb, types: [HANG] testing release v6.11 testing commit 98f7e32f20d28ec452afb208f9cffc08448a2652 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 09c62f2e08fa74bab654383ce328232537bfc0a4afe6391de87b393343915ad2 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in new_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in new_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: no output from test machine run #6: crashed: no output from test machine run #7: OK run #8: OK run #9: OK representative crash: INFO: task hung in del_device_store, types: [HANG UNKNOWN] testing release v6.10 testing commit 0c3836482481200ead7b416ca80c68a29cfdaabd gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 87514af7740ad357afa2ead7e2519501613ae85cd6266e8a32bd21a1ec21a3cb run #0: crashed: INFO: task hung in new_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in new_device_store run #4: crashed: INFO: task hung in linkwatch_event run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK representative crash: INFO: task hung in new_device_store, types: [HANG] testing release v6.8 testing commit e8f897f4afef0031fe618a8e94127a0934896aba gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 90c3180f1abdb838f19fa0bd64f850f38605e6f31bddba8ab8b1d686230eaf42 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in new_device_store run #2: crashed: INFO: task hung in new_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in new_device_store run #5: crashed: no output from test machine run #6: OK run #7: OK run #8: OK run #9: OK representative crash: INFO: task hung in del_device_store, types: [HANG] testing release v6.6 testing commit ffc253263a1375a65fa6c9f62a893e9767fbebfa gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 20a6071f847db9c9c072448ce2ca39cae4992ba43b7af6c3aeac9ecfca67eb8b run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in linkwatch_event run #6: crashed: no output from test machine run #7: OK run #8: OK run #9: OK representative crash: INFO: task hung in del_device_store, types: [HANG] testing release v6.4 testing commit 6995e2de6891c724bfeb2db33d7b87775f913ad1 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 239dcb91c3df81111176d8bf86a8de462f8e37ed80540bc4872070d4b4934fab all runs: OK false negative chance: 0.000 # git bisect start ffc253263a1375a65fa6c9f62a893e9767fbebfa 6995e2de6891c724bfeb2db33d7b87775f913ad1 Bisecting: 14974 revisions left to test after this (roughly 14 steps) [692f5510159c79bfa312a4e27a15e266232bfb4c] Merge tag 'asoc-v6.6' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus testing commit 692f5510159c79bfa312a4e27a15e266232bfb4c gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6580ed2e32593fd0450941836dad06bc5026c686412dd84cb8af38c4886905c9 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: OK run #3: OK run #4: OK run #5: OK run #6: crashed: no output from test machine run #7: OK run #8: OK run #9: OK representative crash: INFO: task hung in del_device_store, types: [HANG UNKNOWN] # git bisect bad 692f5510159c79bfa312a4e27a15e266232bfb4c Bisecting: 7678 revisions left to test after this (roughly 13 steps) [b775d6c5859affe00527cbe74263de05cfe6b9f9] Merge tag 'mips_6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux testing commit b775d6c5859affe00527cbe74263de05cfe6b9f9 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 46995e6896bc967134f4df180cbc7fb8356863720846c1011489978dbf160a79 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: INFO: task hung in del_device_store run #8: crashed: INFO: task hung in del_device_store run #9: crashed: INFO: task hung in new_device_store run #10: crashed: no output from test machine run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad b775d6c5859affe00527cbe74263de05cfe6b9f9 Bisecting: 3061 revisions left to test after this (roughly 12 steps) [3a8a670eeeaa40d87bd38a587438952741980c18] Merge tag 'net-next-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next testing commit 3a8a670eeeaa40d87bd38a587438952741980c18 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a68847814eeb9d96487f2fd2f438347c713a753fa656700a0a4d3e7222315f91 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in del_device_store run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad 3a8a670eeeaa40d87bd38a587438952741980c18 Bisecting: 2103 revisions left to test after this (roughly 11 steps) [6e17c6de3ddf3073741d9c91a796ee696914d8a0] Merge tag 'mm-stable-2023-06-24-19-15' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm testing commit 6e17c6de3ddf3073741d9c91a796ee696914d8a0 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: aa3991091737b55c7cbcee903dcfe693bde0a76afd54e605ea994e25c580e1e5 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in new_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in new_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: INFO: task hung in del_device_store run #8: crashed: INFO: task hung in del_device_store run #9: crashed: no output from test machine run #10: crashed: INFO: task hung in addrconf_dad_work run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: crashed: no output from test machine representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad 6e17c6de3ddf3073741d9c91a796ee696914d8a0 Bisecting: 1048 revisions left to test after this (roughly 10 steps) [2605e80d3438c77190f55b821c6575048c68268e] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux testing commit 2605e80d3438c77190f55b821c6575048c68268e gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ee8c6bcb5db55090a0c671c57015f626c79ebd890cc0f3b8ff92ad068933f563 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: no output from test machine run #4: crashed: no output from test machine run #5: crashed: no output from test machine run #6: crashed: no output from test machine run #7: crashed: no output from test machine run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: crashed: no output from test machine run #14: OK run #15: OK run #16: OK run #17: OK run #18: crashed: INFO: task hung in rtnetlink_rcv_msg run #19: OK representative crash: no output from test machine, types: [UNKNOWN] # git bisect bad 2605e80d3438c77190f55b821c6575048c68268e Bisecting: 665 revisions left to test after this (roughly 9 steps) [0aa69d53ac7c30f6184f88f2e310d808b32b35a5] Merge tag 'for-6.5/io_uring-2023-06-23' of git://git.kernel.dk/linux testing commit 0aa69d53ac7c30f6184f88f2e310d808b32b35a5 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3d8e2bec49ae0913c2dcf3123d9c5690c283153ccea41a8a05b39ed1985d4f47 all runs: OK false negative chance: 0.000 # git bisect good 0aa69d53ac7c30f6184f88f2e310d808b32b35a5 Bisecting: 341 revisions left to test after this (roughly 8 steps) [7cffdbe3607a6cc2dc02d135e13732ec36bc4e28] Merge tag 'x86-boot-2023-06-26' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 7cffdbe3607a6cc2dc02d135e13732ec36bc4e28 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 213a9337f9bf1ffda27f3f6d1e291418334f25ced9704727f9f2070dd9cef679 run #0: crashed: lost connection to test machine run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: INFO: task hung in del_device_store run #8: crashed: INFO: task hung in del_device_store run #9: crashed: INFO: task hung in nsim_drv_remove run #10: crashed: INFO: task hung in nsim_drv_remove run #11: crashed: no output from test machine run #12: crashed: no output from test machine run #13: crashed: no output from test machine run #14: crashed: no output from test machine run #15: crashed: no output from test machine run #16: OK run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in del_device_store, types: [HANG UNKNOWN] # git bisect bad 7cffdbe3607a6cc2dc02d135e13732ec36bc4e28 Bisecting: 161 revisions left to test after this (roughly 7 steps) [658afed19ceed54a52b9e9e69c0791c8868ff55d] mtd: block: use a simple bool to track open for write testing commit 658afed19ceed54a52b9e9e69c0791c8868ff55d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: b2a276b101127efeccf5311779d835c3a1ba26497dded49e44d48d90d0a6f560 all runs: OK false negative chance: 0.000 # git bisect good 658afed19ceed54a52b9e9e69c0791c8868ff55d Bisecting: 80 revisions left to test after this (roughly 6 steps) [0b24be4691c9e6ea13ca70050d42a9f9032fa788] splice: don't call file_accessed in copy_splice_read testing commit 0b24be4691c9e6ea13ca70050d42a9f9032fa788 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 5da4a550a9ecb161b62565e728199978d3999e7b29b1801858b35c2af6ad8397 all runs: OK false negative chance: 0.000 # git bisect good 0b24be4691c9e6ea13ca70050d42a9f9032fa788 Bisecting: 48 revisions left to test after this (roughly 5 steps) [f121ab7f4ac32ed2aa51035534926f9507a8308b] Merge tag 'irqchip-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms into irq/core testing commit f121ab7f4ac32ed2aa51035534926f9507a8308b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 60984942231dab25e6806aa221438c38a943771535e6d39275cb2615cdc0b79d run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: crashed: INFO: task hung in addrconf_dad_work run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad f121ab7f4ac32ed2aa51035534926f9507a8308b Bisecting: 15 revisions left to test after this (roughly 4 steps) [e0b78e9fa34102ce82ef03c71eb13f0b79954ba3] Merge branch irq/loongarch-fixes-6.5 into irq/irqchip-next testing commit e0b78e9fa34102ce82ef03c71eb13f0b79954ba3 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 503aaa8a80f7a2255edbc2c0780b0d948ad40cd41b39e61c399edafd8562167a all runs: OK false negative chance: 0.000 # git bisect good e0b78e9fa34102ce82ef03c71eb13f0b79954ba3 Bisecting: 7 revisions left to test after this (roughly 3 steps) [48f31e496488a25f443c0df52464da446fb1d10c] irqchip/stm32-exti: Fix warning on initialized field overwritten testing commit 48f31e496488a25f443c0df52464da446fb1d10c gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 70cfdcaaa4047486a1f63c19bdc1640c18859b06957fc4079ed5ac5e4d346476 all runs: OK false negative chance: 0.000 # git bisect good 48f31e496488a25f443c0df52464da446fb1d10c Bisecting: 3 revisions left to test after this (roughly 2 steps) [721255b9826bd11c7a38b585905fc2dd0fb94e52] genirq: Use a maple tree for interrupt descriptor management testing commit 721255b9826bd11c7a38b585905fc2dd0fb94e52 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 49a025d5977de3a346de979e762a4e59c7cb68edcd8fb910576095498820a085 run #0: crashed: INFO: task hung in devlink_free run #1: crashed: INFO: task hung in devlink_free run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in devlink_free run #6: crashed: no output from test machine run #7: crashed: no output from test machine run #8: crashed: no output from test machine run #9: crashed: no output from test machine run #10: crashed: no output from test machine run #11: OK run #12: OK run #13: crashed: no output from test machine run #14: crashed: no output from test machine run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK representative crash: no output from test machine, types: [UNKNOWN] # git bisect bad 721255b9826bd11c7a38b585905fc2dd0fb94e52 Bisecting: 1 revision left to test after this (roughly 1 step) [bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db] genirq: Use hlist for managing resend handlers testing commit bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: fa5904cf2fea177af837c24f1412ac96a810bf54f33a3232e77dc77c09dc3dc3 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in devlink_free run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: no output from test machine run #5: crashed: no output from test machine run #6: crashed: no output from test machine run #7: OK run #8: crashed: no output from test machine run #9: OK run #10: OK run #11: crashed: INFO: task hung in devlink_free run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: crashed: no output from test machine representative crash: INFO: task hung in del_device_store, types: [HANG UNKNOWN] # git bisect bad bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db Bisecting: 0 revisions left to test after this (roughly 0 steps) [d15121be7485655129101f3960ae6add40204463] Revert "softirq: Let ksoftirqd do its job" testing commit d15121be7485655129101f3960ae6add40204463 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ef717219c1f9b67ef17f9e051761e7478312091e6e2c22100ddab5671a30f8a8 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in register_netdevice_notifier run #2: crashed: INFO: task hung in devlink_free run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in wg_netns_pre_exit run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in linkwatch_event run #7: crashed: no output from test machine run #8: crashed: no output from test machine run #9: crashed: no output from test machine run #10: crashed: INFO: task hung in devlink_free run #11: OK run #12: OK run #13: OK run #14: OK run #15: crashed: no output from test machine run #16: OK run #17: OK run #18: OK run #19: crashed: no output from test machine representative crash: INFO: task hung in del_device_store, types: [HANG UNKNOWN] # git bisect bad d15121be7485655129101f3960ae6add40204463 d15121be7485655129101f3960ae6add40204463 is the first bad commit commit d15121be7485655129101f3960ae6add40204463 Author: Paolo Abeni Date: Mon May 8 08:17:44 2023 +0200 Revert "softirq: Let ksoftirqd do its job" This reverts the following commits: 4cd13c21b207 ("softirq: Let ksoftirqd do its job") 3c53776e29f8 ("Mark HI and TASKLET softirq synchronous") 1342d8080f61 ("softirq: Don't skip softirq execution when softirq thread is parking") in a single change to avoid known bad intermediate states introduced by a patch series reverting them individually. Due to the mentioned commit, when the ksoftirqd threads take charge of softirq processing, the system can experience high latencies. In the past a few workarounds have been implemented for specific side-effects of the initial ksoftirqd enforcement commit: commit 1ff688209e2e ("watchdog: core: make sure the watchdog_worker is not deferred") commit 8d5755b3f77b ("watchdog: softdog: fire watchdog even if softirqs do not get to run") commit 217f69743681 ("net: busy-poll: allow preemption in sk_busy_loop()") commit 3c53776e29f8 ("Mark HI and TASKLET softirq synchronous") But the latency problem still exists in real-life workloads, see the link below. The reverted commit intended to solve a live-lock scenario that can now be addressed with the NAPI threaded mode, introduced with commit 29863d41bb6e ("net: implement threaded-able napi poll loop support"), which is nowadays in a pretty stable status. While a complete solution to put softirq processing under nice resource control would be preferable, that has proven to be a very hard task. In the short term, remove the main pain point, and also simplify a bit the current softirq implementation. Signed-off-by: Paolo Abeni Signed-off-by: Thomas Gleixner Tested-by: Jason Xing Reviewed-by: Jakub Kicinski Reviewed-by: Eric Dumazet Reviewed-by: Sebastian Andrzej Siewior Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: netdev@vger.kernel.org Link: https://lore.kernel.org/netdev/305d7742212cbe98621b16be782b0562f1012cb6.camel@redhat.com Link: https://lore.kernel.org/r/57e66b364f1b6f09c9bc0316742c3b14f4ce83bd.1683526542.git.pabeni@redhat.com kernel/softirq.c | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-) accumulated error probability: 0.00 parent commit ac9a78681b921877518763ba0e89202254349d1b wasn't tested testing commit ac9a78681b921877518763ba0e89202254349d1b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 931a91d1e381e0d51e6f7e373d2f5002a18d0daafc5ce6784e4df0b45c0909ed culprit signature: ef717219c1f9b67ef17f9e051761e7478312091e6e2c22100ddab5671a30f8a8 parent signature: 931a91d1e381e0d51e6f7e373d2f5002a18d0daafc5ce6784e4df0b45c0909ed reproducer is flaky (0.39 repro chance estimate) revisions tested: 27, total time: 8h16m7.120502516s (build: 4h0m40.837125766s, test: 3h59m48.010198799s) first bad commit: d15121be7485655129101f3960ae6add40204463 Revert "softirq: Let ksoftirqd do its job" recipients (to): ["bigeasy@linutronix.de" "edumazet@google.com" "kerneljasonxing@gmail.com" "kuba@kernel.org" "pabeni@redhat.com" "tglx@linutronix.de"] recipients (cc): [] crash: INFO: task hung in del_device_store INFO: task syz-executor:5829 blocked for more than 143 seconds. Not tainted 6.4.0-rc1-syzkaller #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:syz-executor state:D stack:10808 pid:5829 ppid:1 flags:0x00004004 Call Trace: context_switch kernel/sched/core.c:5343 [inline] __schedule+0x6af/0x9c0 kernel/sched/core.c:6669 schedule+0x68/0xc0 kernel/sched/core.c:6745 schedule_preempt_disabled+0x32/0x60 kernel/sched/core.c:6804 __mutex_lock_common kernel/locking/mutex.c:679 [inline] __mutex_lock+0x449/0x970 kernel/locking/mutex.c:747 del_device_store+0x7d/0x1a0 drivers/net/netdevsim/bus.c:209 kernfs_fop_write_iter+0x217/0x2b0 fs/kernfs/file.c:334 call_write_iter include/linux/fs.h:1868 [inline] new_sync_write fs/read_write.c:491 [inline] vfs_write+0x3ae/0x650 fs/read_write.c:584 ksys_write+0xc6/0x180 fs/read_write.c:637 do_syscall_64+0x46/0xc0 entry_SYSCALL_64_after_hwframe+0x73/0xdd RIP: 0033:0x7fcf0257d2c0 RSP: 002b:00007ffd100c7c18 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007fcf0257d2c0 RDX: 0000000000000001 RSI: 00007ffd100c7c40 RDI: 0000000000000005 RBP: 00007fcf025f2876 R08: 0000000000000001 R09: 00007ffd100c7a47 R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000001 R13: 00007ffd100c7c40 R14: 00007fcf03264620 R15: 0000000000000003 Showing all locks held in the system: 5 locks held by kworker/u4:1/12: #0: ffff888101643d38 ((wq_completion)netns){....}-{0:0}, at: process_one_work+0x26a/0x740 kernel/workqueue.c:2378 #1: ffffc9000006be60 (net_cleanup_work){....}-{0:0}, at: process_one_work+0x294/0x740 kernel/workqueue.c:2380 #2: ffffffff85b3f498 (pernet_ops_rwsem){....}-{3:3}, at: cleanup_net+0x4a/0x5e0 net/core/net_namespace.c:576 #3: ffffffff85b41418 (rtnl_mutex){....}-{3:3}, at: default_device_exit_batch+0x43/0x370 net/core/dev.c:11377 #4: ffffffff857feab8 (rcu_state.exp_mutex){....}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:293 [inline] #4: ffffffff857feab8 (rcu_state.exp_mutex){....}-{3:3}, at: synchronize_rcu_expedited+0x3a0/0x700 kernel/rcu/tree_exp.h:992 1 lock held by rcu_tasks_kthre/13: #0: ffffffff857fd9d8 (rcu_tasks.tasks_gp_mutex){....}-{3:3}, at: rcu_tasks_one_gp kernel/rcu/tasks.h:518 [inline] #0: ffffffff857fd9d8 (rcu_tasks.tasks_gp_mutex){....}-{3:3}, at: rcu_tasks_kthread+0x9c/0x460 kernel/rcu/tasks.h:563 1 lock held by rcu_tasks_trace/14: #0: ffffffff857fdcb8 (rcu_tasks_trace.tasks_gp_mutex){....}-{3:3}, at: rcu_tasks_one_gp kernel/rcu/tasks.h:518 [inline] #0: ffffffff857fdcb8 (rcu_tasks_trace.tasks_gp_mutex){....}-{3:3}, at: rcu_tasks_kthread+0x9c/0x460 kernel/rcu/tasks.h:563 2 locks held by kworker/1:0/22: 1 lock held by khungtaskd/28: #0: ffffffff857fd8b0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x0/0x30 arch/x86/kernel/nmi.c:311 5 locks held by kworker/1:3/3997: 2 locks held by getty/4411: #0: ffff88811713d898 (&tty->ldisc_sem){....}-{0:0}, at: tty_ldisc_ref_wait+0x25/0x50 drivers/tty/tty_ldisc.c:243 #1: ffffc9000d2c72f0 (&ldata->atomic_read_lock){....}-{3:3}, at: n_tty_read+0x319/0xf00 drivers/tty/n_tty.c:2176 6 locks held by syz-executor/5819: #0: ffff888118f67450 (sb_writers#8){....}-{0:0}, at: ksys_write+0xc6/0x180 fs/read_write.c:637 #1: ffff8881192e5e88 (&of->mutex){....}-{3:3}, at: kernfs_fop_write_iter+0x12e/0x2b0 fs/kernfs/file.c:325 #2: ffff888101f7d740 (kn->active#55){....}-{0:0}, at: kernfs_fop_write_iter+0x137/0x2b0 fs/kernfs/file.c:326 #3: ffffffff859e02d0 (nsim_bus_dev_list_lock){....}-{3:3}, at: del_device_store+0x7d/0x1a0 drivers/net/netdevsim/bus.c:209 #4: ffff8881193bc0e8 (&dev->mutex){....}-{3:3}, at: device_lock include/linux/device.h:837 [inline] #4: ffff8881193bc0e8 (&dev->mutex){....}-{3:3}, at: __device_driver_lock drivers/base/dd.c:1095 [inline] #4: ffff8881193bc0e8 (&dev->mutex){....}-{3:3}, at: device_release_driver_internal+0x6b/0x3a0 drivers/base/dd.c:1292 #5: ffffffff85b3f498 (pernet_ops_rwsem){....}-{3:3}, at: unregister_netdevice_notifier+0x34/0x1c0 net/core/dev.c:1757 4 locks held by syz-executor/5829: #0: ffff888118f67450 (sb_writers#8){....}-{0:0}, at: ksys_write+0xc6/0x180 fs/read_write.c:637 #1: ffff888119254088 (&of->mutex){....}-{3:3}, at: kernfs_fop_write_iter+0x12e/0x2b0 fs/kernfs/file.c:325 #2: ffff888101f7d740 (kn->active#55){....}-{0:0}, at: kernfs_fop_write_iter+0x137/0x2b0 fs/kernfs/file.c:326 #3: ffffffff859e02d0 (nsim_bus_dev_list_lock){....}-{3:3}, at: del_device_store+0x7d/0x1a0 drivers/net/netdevsim/bus.c:209 1 lock held by syz-executor/6566: #0: ffffffff85b41418 (rtnl_mutex){....}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:78 [inline] #0: ffffffff85b41418 (rtnl_mutex){....}-{3:3}, at: rtnetlink_rcv_msg+0x5db/0x750 net/core/rtnetlink.c:6392 1 lock held by syz-executor/6569: #0: ffffffff85b41418 (rtnl_mutex){....}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:78 [inline] #0: ffffffff85b41418 (rtnl_mutex){....}-{3:3}, at: rtnetlink_rcv_msg+0x5db/0x750 net/core/rtnetlink.c:6392 1 lock held by syz-executor/6572: #0: ffffffff85b41418 (rtnl_mutex){....}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:78 [inline] #0: ffffffff85b41418 (rtnl_mutex){....}-{3:3}, at: rtnetlink_rcv_msg+0x5db/0x750 net/core/rtnetlink.c:6392 1 lock held by syz-executor/6575: #0: ffffffff85b41418 (rtnl_mutex){....}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:78 [inline] #0: ffffffff85b41418 (rtnl_mutex){....}-{3:3}, at: rtnetlink_rcv_msg+0x5db/0x750 net/core/rtnetlink.c:6392 2 locks held by syz.1.75/6650: #0: ffff888161ae69a8 (&nft_net->commit_mutex){....}-{3:3}, at: nf_tables_valid_genid+0x45/0x90 net/netfilter/nf_tables_api.c:9788 #1: ffffffff857feab8 (rcu_state.exp_mutex){....}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:325 [inline] #1: ffffffff857feab8 (rcu_state.exp_mutex){....}-{3:3}, at: synchronize_rcu_expedited+0x240/0x700 kernel/rcu/tree_exp.h:992 ============================================= NMI backtrace for cpu 0 CPU: 0 PID: 28 Comm: khungtaskd Not tainted 6.4.0-rc1-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0xf4/0x180 lib/dump_stack.c:106 nmi_cpu_backtrace+0x234/0x250 lib/nmi_backtrace.c:113 nmi_trigger_cpumask_backtrace+0xf2/0x210 lib/nmi_backtrace.c:62 trigger_all_cpu_backtrace include/linux/nmi.h:148 [inline] check_hung_uninterruptible_tasks kernel/hung_task.c:222 [inline] watchdog+0x81f/0x860 kernel/hung_task.c:379 kthread+0x113/0x140 kernel/kthread.c:379 ret_from_fork+0x29/0x50 arch/x86/entry/entry_64.S:308 Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 CPU: 1 PID: 3997 Comm: kworker/1:3 Not tainted 6.4.0-rc1-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024 Workqueue: events_power_efficient neigh_periodic_work RIP: 0010:__sanitizer_cov_trace_pc+0x0/0x70 kernel/kcov.c:200 Code: 89 fb e8 23 00 00 00 48 8b 3d 64 e3 97 04 48 89 de 5b e9 03 23 2b 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1e fa 48 8b 04 24 65 48 8b 0d a0 10 cb 7e 65 8b 15 a1 10 cb RSP: 0018:ffffc900000d0850 EFLAGS: 00000206 RAX: ffff888105ef5280 RBX: ffffc900000d08f0 RCX: ffff888105ef5280 RDX: 0000000080000101 RSI: 0000000000000000 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffffffff83c2e56f R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881795d1b00 R13: ffff888117784700 R14: 0000000000000000 R15: ffff8881795d1b00 FS: 0000000000000000(0000) GS:ffff888237d00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000200000c0 CR3: 000000011951e000 CR4: 00000000003506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: rcu_lock_acquire+0x5/0x40 include/linux/rcupdate.h:326 rcu_read_lock include/linux/rcupdate.h:773 [inline] ip_route_output_key_hash net/ipv4/route.c:2640 [inline] __ip_route_output_key include/net/route.h:134 [inline] ip_route_output_flow+0x9f/0x140 net/ipv4/route.c:2869 ip_route_output_key include/net/route.h:150 [inline] ip_route_me_harder+0x372/0x6b0 net/ipv4/netfilter.c:52 synproxy_send_tcp+0x178/0x240 net/netfilter/nf_synproxy_core.c:431 synproxy_send_client_synack+0x304/0x330 net/netfilter/nf_synproxy_core.c:484 nft_synproxy_eval_v4+0x184/0x1d0 net/netfilter/nft_synproxy.c:59 nft_synproxy_do_eval+0x170/0x370 net/netfilter/nft_synproxy.c:141 expr_call_ops_eval net/netfilter/nf_tables_core.c:240 [inline] nft_do_chain+0x28a/0xab0 net/netfilter/nf_tables_core.c:288 nft_do_chain_inet+0x144/0x170 net/netfilter/nft_chain_filter.c:161 nf_hook_entry_hookfn include/linux/netfilter.h:143 [inline] nf_hook_slow+0x54/0x140 net/netfilter/core.c:626 nf_hook include/linux/netfilter.h:258 [inline] NF_HOOK include/linux/netfilter.h:301 [inline] ip_local_deliver+0x12e/0x170 net/ipv4/ip_input.c:254 NF_HOOK include/linux/netfilter.h:303 [inline] ip_rcv+0x55/0x130 net/ipv4/ip_input.c:569 __netif_receive_skb_one_core net/core/dev.c:5491 [inline] __netif_receive_skb+0xe4/0x210 net/core/dev.c:5605 process_backlog+0x13a/0x210 net/core/dev.c:5933 __napi_poll+0x44/0x320 net/core/dev.c:6496 napi_poll net/core/dev.c:6563 [inline] net_rx_action+0x1f7/0x4a0 net/core/dev.c:6696 __do_softirq+0x110/0x2fa kernel/softirq.c:553 do_softirq+0x86/0xf0 kernel/softirq.c:454 __local_bh_enable_ip+0x69/0x70 kernel/softirq.c:381 neigh_periodic_work+0x387/0x400 net/core/neighbour.c:1006 process_one_work+0x319/0x740 kernel/workqueue.c:2405 worker_thread+0x328/0x580 kernel/workqueue.c:2552 kthread+0x113/0x140 kernel/kthread.c:379 ret_from_fork+0x29/0x50 arch/x86/entry/entry_64.S:308