ci starts bisection 2025-01-27 04:56:20.710091519 +0000 UTC m=+196778.629681025 bisecting cause commit starting from feffde684ac29a3b7aec82d2df850fbdbdee55e4 building syzkaller on b50eb251af3b122fb1b2c574dde0c3d16f6a8cfd ensuring issue is reproducible on original commit feffde684ac29a3b7aec82d2df850fbdbdee55e4 testing commit feffde684ac29a3b7aec82d2df850fbdbdee55e4 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 4b1df208c0e2e0bc35670d1b9f77e882d659d75b7383b258ff9709531792b2c5 run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: rcu detected stall in sys_unlink run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in corrupted run #4: crashed: INFO: task hung in corrupted run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: rcu detected stall in corrupted run #7: crashed: INFO: task hung in corrupted run #8: crashed: INFO: rcu detected stall in sys_pipe2 run #9: crashed: INFO: task hung in corrupted run #10: crashed: INFO: rcu detected stall in corrupted run #11: crashed: INFO: rcu detected stall in worker_thread run #12: crashed: INFO: task hung in corrupted run #13: crashed: INFO: task hung in corrupted run #14: crashed: INFO: rcu detected stall in corrupted run #15: crashed: INFO: rcu detected stall in corrupted run #16: crashed: INFO: task hung in corrupted run #17: crashed: WARNING in rcu_check_gp_start_stall run #18: crashed: INFO: task hung in corrupted run #19: crashed: INFO: task hung in corrupted representative crash: INFO: rcu detected stall in corrupted, types: [HANG] check whether we can drop unnecessary instrumentation disabling configs for [BUG KASAN LOCKDEP ATOMIC_SLEEP LEAK UBSAN], they are not needed testing commit feffde684ac29a3b7aec82d2df850fbdbdee55e4 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6c29a0150eee8afceb86b9fffdf7c0cd10544c6f2a4872ec7d10b0b2335cadd2 run #0: crashed: INFO: task hung in corrupted run #1: crashed: INFO: task hung in corrupted run #2: crashed: INFO: task hung in corrupted run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: task hung in corrupted run #6: crashed: INFO: task hung in corrupted run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: INFO: task hung in corrupted run #9: crashed: INFO: task hung in corrupted representative crash: INFO: task hung in corrupted, types: [HANG] the bug reproduces without the instrumentation disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed kconfig minimization: base=4045 full=8243 leaves diff=2124 split chunks (needed=false): <2124> split chunk #0 of len 2124 into 5 parts testing without sub-chunk 1/5 disabling configs for [LOCKDEP ATOMIC_SLEEP LEAK UBSAN BUG KASAN], they are not needed testing commit feffde684ac29a3b7aec82d2df850fbdbdee55e4 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: b743da6659d50b95ac95471a3e6fd7018e322bedd78beac1b878ba395e0cd644 all runs: OK false negative chance: 0.000 testing without sub-chunk 2/5 disabling configs for [LOCKDEP ATOMIC_SLEEP LEAK UBSAN BUG KASAN], they are not needed testing commit feffde684ac29a3b7aec82d2df850fbdbdee55e4 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 60dd8370fe07f7297e42f544839f3f4918d7c15e565f61dff61b4401d7184c13 all runs: OK false negative chance: 0.000 testing without sub-chunk 3/5 disabling configs for [LEAK UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit feffde684ac29a3b7aec82d2df850fbdbdee55e4 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ed09dc68d0586a3ceead460f2c9b5cf4c3cffdd39dddf64e6a75d20226b99cc2 all runs: OK false negative chance: 0.000 testing without sub-chunk 4/5 disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP LEAK UBSAN BUG], they are not needed testing commit feffde684ac29a3b7aec82d2df850fbdbdee55e4 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 57fb213ba57ac5a611cae403d0a4f4683a6cf0b52201b247a015fd20e23a4a3f run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: task hung in addrconf_dad_work run #6: crashed: INFO: rcu detected stall in corrupted run #7: crashed: INFO: task hung in reg_check_chans_work run #8: crashed: no output from test machine run #9: crashed: no output from test machine representative crash: INFO: rcu detected stall in corrupted, types: [HANG] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP LEAK], they are not needed testing commit feffde684ac29a3b7aec82d2df850fbdbdee55e4 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6d6427ca28dacb90e2f1465b801f119bcbd2d2f0fde1da9cd48a4ab167f8305c all runs: OK false negative chance: 0.000 minimized to 1699 configs; suspects: [6LOWPAN 6LOWPAN_GHC_EXT_HDR_DEST 6LOWPAN_GHC_EXT_HDR_FRAG 6LOWPAN_GHC_EXT_HDR_HOP 6LOWPAN_GHC_EXT_HDR_ROUTE 6LOWPAN_GHC_ICMPV6 6LOWPAN_GHC_UDP 6LOWPAN_NHC 6LOWPAN_NHC_DEST 6LOWPAN_NHC_FRAGMENT 6LOWPAN_NHC_HOP 6LOWPAN_NHC_IPV6 6LOWPAN_NHC_MOBILITY 6LOWPAN_NHC_ROUTING 6LOWPAN_NHC_UDP 6PACK 842_COMPRESS 842_DECOMPRESS 9P_FSCACHE 9P_FS_POSIX_ACL 9P_FS_SECURITY ACORN_PARTITION ACORN_PARTITION_ADFS ACORN_PARTITION_CUMANA ACORN_PARTITION_EESOX ACORN_PARTITION_ICS ACORN_PARTITION_POWERTEC ACORN_PARTITION_RISCIX ACPI_EC ACPI_NFIT ACPI_NHLT ACPI_PLATFORM_PROFILE ADFS_FS AFFS_FS AFS_FS AFS_FSCACHE AF_KCM AF_RXRPC AF_RXRPC_IPV6 AIX_PARTITION AMD_SFH_HID AMIGA_PARTITION ANDROID_BINDERFS ANDROID_BINDER_IPC ANON_VMA_NAME APERTURE_HELPERS APPLE_MFI_FASTCHARGE AR5523 ARCH_DEFAULT_CRASH_DUMP ARCH_ENABLE_MEMORY_HOTREMOVE ARCH_ENABLE_THP_MIGRATION ARCH_HAS_EXECMEM_ROX ARCH_HAS_PREEMPT_LAZY ARCH_HAS_USER_SHADOW_STACK ARCH_SUPPORTS_AUTOFDO_CLANG ARCH_SUPPORTS_HUGE_PFNMAP ARCH_SUPPORTS_PMD_PFNMAP ARCH_SUPPORTS_PROPELLER_CLANG ARCH_SUPPORTS_PUD_PFNMAP ARCH_SUPPORTS_RT ARCH_WANT_PMD_MKWRITE ASM_MODVERSIONS ASUS_TF103C_DOCK ASUS_WMI ASYNC_CORE ASYNC_MEMCPY ASYNC_PQ ASYNC_RAID6_RECOV ASYNC_TX_DMA ASYNC_XOR ATARI_PARTITION ATA_GENERIC ATA_OVER_ETH ATH10K ATH10K_CE ATH10K_LEDS ATH10K_PCI ATH10K_USB ATH11K ATH6KL ATH6KL_USB ATH9K ATH9K_AHB ATH9K_BTCOEX_SUPPORT ATH9K_CHANNEL_CONTEXT ATH9K_COMMON ATH9K_COMMON_DEBUG ATH9K_COMMON_SPECTRAL ATH9K_DEBUGFS ATH9K_DYNACK ATH9K_HTC ATH9K_HTC_DEBUGFS ATH9K_HW ATH9K_PCI ATH9K_PCOEM ATH9K_RFKILL ATH_COMMON ATM ATM_BR2684 ATM_CLIP ATM_DRIVERS ATM_LANE ATM_MPOA ATM_TCP AUXILIARY_BUS AX25 AX25_DAMA_SLAVE AX88796B_PHY BAREUDP BATMAN_ADV BATMAN_ADV_BATMAN_V BATMAN_ADV_BLA BATMAN_ADV_DAT BATMAN_ADV_MCAST BATMAN_ADV_NC BCACHE BCACHEFS_DEBUG BCACHEFS_ERASURE_CODING BCACHEFS_FS BCACHEFS_POSIX_ACL BCACHEFS_QUOTA BCACHEFS_SIX_OPTIMISTIC_SPIN BCMA BCMA_HOST_PCI_POSSIBLE BEFS_FS BFQ_CGROUP_DEBUG BFQ_GROUP_IOSCHED BFS_FS BIG_KEYS BLK_CGROUP_PUNT_BIO BLK_CGROUP_RWSTAT BLK_DEV_BSGLIB BLK_DEV_INTEGRITY BLK_DEV_NBD BLK_DEV_NULL_BLK BLK_DEV_NULL_BLK_FAULT_INJECTION BLK_DEV_NVME BLK_DEV_PMEM BLK_DEV_RAM BLK_DEV_RNBD BLK_DEV_RNBD_CLIENT BLK_DEV_THROTTLING BLK_DEV_ZONED BLK_ICQ BLK_INLINE_ENCRYPTION BLK_INLINE_ENCRYPTION_FALLBACK BLK_WBT BLK_WBT_MQ BONDING BOOT_VESA_SUPPORT BPF_EVENTS BPF_JIT BPF_JIT_ALWAYS_ON BPF_JIT_DEFAULT_ON BPF_LSM BPF_PRELOAD BPF_PRELOAD_UMD BPF_STREAM_PARSER BPF_SYSCALL BPQETHER BRIDGE BRIDGE_CFM BRIDGE_EBT_802_3 BRIDGE_EBT_AMONG BRIDGE_EBT_ARP BRIDGE_EBT_ARPREPLY BRIDGE_EBT_BROUTE BRIDGE_EBT_DNAT BRIDGE_EBT_IP BRIDGE_EBT_IP6 BRIDGE_EBT_LIMIT BRIDGE_EBT_LOG BRIDGE_EBT_MARK BRIDGE_EBT_MARK_T BRIDGE_EBT_NFLOG BRIDGE_EBT_PKTTYPE BRIDGE_EBT_REDIRECT BRIDGE_EBT_SNAT BRIDGE_EBT_STP BRIDGE_EBT_T_FILTER BRIDGE_EBT_T_NAT BRIDGE_EBT_VLAN BRIDGE_IGMP_SNOOPING BRIDGE_MRP BRIDGE_NETFILTER BRIDGE_NF_EBTABLES BRIDGE_NF_EBTABLES_LEGACY BRIDGE_VLAN_FILTERING BSD_DISKLABEL BSD_PROCESS_ACCT_V3 BT BTRFS_ASSERT BTRFS_FS BTRFS_FS_POSIX_ACL BTRFS_FS_REF_VERIFY BTT BT_6LOWPAN BT_ATH3K BT_BCM BT_BNEP BT_BNEP_MC_FILTER BT_BNEP_PROTO_FILTER BT_BREDR BT_HCIBCM203X BT_HCIBFUSB BT_HCIBPA10X BT_HCIBTUSB BT_HCIBTUSB_AUTOSUSPEND BT_HCIBTUSB_BCM BT_HCIBTUSB_MTK BT_HCIBTUSB_POLL_SYNC BT_HCIBTUSB_RTL BT_HCIUART BT_HCIUART_3WIRE BT_HCIUART_AG6XX BT_HCIUART_BCSP BT_HCIUART_H4 BT_HCIUART_LL BT_HCIUART_MRVL BT_HCIUART_QCA BT_HCIUART_SERDEV BT_HCIVHCI BT_INTEL BT_LE BT_LEDS BT_LE_L2CAP_ECRED BT_MRVL BT_MRVL_SDIO BT_MSFTEXT BT_MTK BT_MTKSDIO BT_MTKUART BT_QCA BT_RFCOMM BT_RFCOMM_TTY BT_RTL CACHEFILES CAIF CAIF_DEBUG CAIF_DRIVERS CAIF_NETDEV CAIF_TTY CAIF_USB CAIF_VIRTIO CAN CAN_8DEV_USB CAN_BCM CAN_CALC_BITTIMING CAN_DEV CAN_EMS_USB CAN_ESD_USB CAN_ETAS_ES58X CAN_F81604 CAN_GS_USB CAN_GW CAN_IFI_CANFD CAN_ISOTP CAN_J1939 CAN_KVASER_USB CAN_MCBA_USB CAN_NETLINK CAN_PEAK_USB CAN_RAW CAN_RX_OFFLOAD CAN_SLCAN CAN_UCAN CAN_VCAN CAN_VXCAN CARDBUS CARL9170 CARL9170_HWRNG CARL9170_LEDS CARL9170_WPC CEC_CORE CEPH_FS CEPH_FSCACHE CEPH_FS_POSIX_ACL CEPH_LIB CEPH_LIB_USE_DNS_RESOLVER CFG80211 CFG80211_CRDA_SUPPORT CFG80211_DEBUGFS CFG80211_DEFAULT_PS CFG80211_REQUIRE_SIGNED_REGDB CFG80211_USE_KERNEL_REGDB_KEYS CFG80211_WEXT CFS_BANDWIDTH CGROUP_BPF CHARGER_BQ24190 CHARGER_ISP1704 CHECKPOINT_RESTORE CHR_DEV_ST CIFS CIFS_ALLOW_INSECURE_LEGACY CIFS_DEBUG CIFS_DFS_UPCALL CIFS_FSCACHE CIFS_POSIX CIFS_SMB_DIRECT CIFS_SWN_UPCALL CIFS_UPCALL CIFS_XATTR CLOSURES CLS_U32_MARK CLS_U32_PERF CMA CMA_SIZE_SEL_MAX CMDLINE_PARTITION COMEDI COMEDI_DT9812 COMEDI_NI_USB6501 COMEDI_USBDUX COMEDI_USBDUXFAST COMEDI_USBDUXSIGMA COMEDI_USB_DRIVERS COMEDI_VMK80XX COMMON_CLK COMPAT_NETLINK_MESSAGES CONTIG_ALLOC COUNTER CRAMFS CRAMFS_BLOCKDEV CRAMFS_MTD CRC4 CRC64 CRC64_ROCKSOFT CRC7 CRC8 CRC_ITU_T CRC_T10DIF CRYPTO_842 CRYPTO_ADIANTUM CRYPTO_AEGIS128 CRYPTO_AEGIS128_AESNI_SSE2 CRYPTO_AES_NI_INTEL CRYPTO_AES_TI CRYPTO_ANSI_CPRNG CRYPTO_ANUBIS CRYPTO_ARC4 CRYPTO_ARCH_HAVE_LIB_BLAKE2S CRYPTO_ARCH_HAVE_LIB_CHACHA CRYPTO_ARCH_HAVE_LIB_CURVE25519 CRYPTO_ARCH_HAVE_LIB_POLY1305 CRYPTO_ARIA CRYPTO_ARIA_AESNI_AVX_X86_64 CRYPTO_BLAKE2B CRYPTO_BLAKE2S_X86 CRYPTO_BLOWFISH CRYPTO_BLOWFISH_COMMON CRYPTO_BLOWFISH_X86_64 CRYPTO_CAMELLIA CRYPTO_CAMELLIA_AESNI_AVX2_X86_64 CRYPTO_CAMELLIA_AESNI_AVX_X86_64 CRYPTO_CAMELLIA_X86_64 CRYPTO_CAST5 CRYPTO_CAST5_AVX_X86_64 CRYPTO_CAST6 CRYPTO_CAST6_AVX_X86_64 CRYPTO_CAST_COMMON CRYPTO_CHACHA20POLY1305 CRYPTO_CHACHA20_X86_64 CRYPTO_CRC32 CRYPTO_CRC32C_INTEL CRYPTO_CRC32_PCLMUL CRYPTO_CRC64_ROCKSOFT CRYPTO_CRCT10DIF CRYPTO_CRCT10DIF_PCLMUL CRYPTO_CRYPTD CRYPTO_CURVE25519 CRYPTO_CURVE25519_X86 CRYPTO_DEFLATE CRYPTO_DES CRYPTO_DES3_EDE_X86_64 CRYPTO_DEV_CCP CRYPTO_DEV_CCP_DD CRYPTO_DEV_PADLOCK CRYPTO_DEV_PADLOCK_AES CRYPTO_DEV_PADLOCK_SHA CRYPTO_DEV_QAT CRYPTO_DEV_QAT_C3XXX CRYPTO_DEV_QAT_C3XXXVF CRYPTO_DEV_QAT_C62X CRYPTO_DEV_QAT_C62XVF CRYPTO_DEV_QAT_DH895xCC CRYPTO_DEV_QAT_DH895xCCVF CRYPTO_DEV_VIRTIO CRYPTO_DH CRYPTO_DRBG_CTR CRYPTO_DRBG_HASH CRYPTO_ECDH CRYPTO_ECRDSA CRYPTO_ENGINE CRYPTO_ESSIV CRYPTO_FCRYPT CRYPTO_GHASH_CLMUL_NI_INTEL CRYPTO_HCTR2 CRYPTO_KDF800108_CTR CRYPTO_KEYWRAP CRYPTO_KHAZAD CRYPTO_KPP CRYPTO_LIB_ARC4 CRYPTO_LIB_CHACHA CRYPTO_LIB_CHACHA20POLY1305 CRYPTO_LIB_CURVE25519 CRYPTO_LIB_CURVE25519_GENERIC CRYPTO_LIB_POLY1305 CRYPTO_LIB_POLY1305_GENERIC CRYPTO_LRW CRYPTO_LZ4 CRYPTO_LZ4HC CRYPTO_NHPOLY1305 CRYPTO_NHPOLY1305_AVX2 CRYPTO_NHPOLY1305_SSE2 CRYPTO_PCBC CRYPTO_PCRYPT CRYPTO_POLY1305 CRYPTO_POLY1305_X86_64 CRYPTO_POLYVAL CRYPTO_POLYVAL_CLMUL_NI CRYPTO_RMD160 CRYPTO_SEED CRYPTO_SERPENT CRYPTO_SERPENT_AVX2_X86_64 CRYPTO_SERPENT_AVX_X86_64 CRYPTO_SERPENT_SSE2_X86_64 CRYPTO_SHA1_SSSE3 CRYPTO_SHA256_SSSE3 CRYPTO_SHA512_SSSE3 CRYPTO_SIMD CRYPTO_SM3_AVX_X86_64 CRYPTO_SM4_AESNI_AVX2_X86_64 CRYPTO_SM4_AESNI_AVX_X86_64 CRYPTO_STREEBOG CRYPTO_TEA CRYPTO_TWOFISH CRYPTO_TWOFISH_AVX_X86_64 CRYPTO_TWOFISH_COMMON CRYPTO_TWOFISH_X86_64 CRYPTO_TWOFISH_X86_64_3WAY CRYPTO_USER CRYPTO_USER_API CRYPTO_USER_API_AEAD CRYPTO_USER_API_ENABLE_OBSOLETE CRYPTO_USER_API_HASH CRYPTO_USER_API_RNG CRYPTO_USER_API_SKCIPHER CRYPTO_VMAC CRYPTO_WP512 CRYPTO_XCTR CRYPTO_XXHASH CRYPTO_ZSTD CUSE CYPRESS_FIRMWARE DAMON DAMON_PADDR DAMON_RECLAIM DAMON_VADDR DAX DCA DCB DEFAULT_CODEL DEVICE_MIGRATION DEVICE_PRIVATE DEV_COREDUMP DEV_DAX DLM DLN2_ADC DMABUF_HEAPS DMABUF_HEAPS_CMA DMABUF_HEAPS_SYSTEM DMABUF_MOVE_NOTIFY DMA_CMA DMA_ENGINE_RAID DM_AUDIT DM_BIO_PRISON DM_BUFIO DM_CACHE DM_CACHE_SMQ DM_CLONE DM_CRYPT DM_FLAKEY DM_INTEGRITY DM_MULTIPATH DM_MULTIPATH_QL DM_MULTIPATH_ST DM_PERSISTENT_DATA DM_RAID DM_SNAPSHOT DM_THIN_PROVISIONING DM_UEVENT DM_VERITY DM_VERITY_FEC DM_WRITECACHE DM_ZONED DRAGONRISE_FF DRM DRM_AUX_BRIDGE DRM_BOCHS DRM_BRIDGE DRM_BUDDY DRM_CIRRUS_QEMU DRM_CLIENT DRM_CLIENT_LIB DRM_CLIENT_SELECTION DRM_CLIENT_SETUP DRM_DEBUG_MM DRM_DISPLAY_DP_AUX_BUS DRM_DISPLAY_DP_HELPER DRM_DISPLAY_DSC_HELPER DRM_DISPLAY_HDCP_HELPER DRM_DISPLAY_HDMI_HELPER DRM_DISPLAY_HELPER DRM_FBDEV_EMULATION DRM_GEM_SHMEM_HELPER DRM_GM12U320 DRM_GUD DRM_I915 DRM_I915_CAPTURE_ERROR DRM_I915_COMPRESS_ERROR DRM_I915_USERPTR DRM_KMS_HELPER DRM_MIPI_DSI DRM_PANEL DRM_PANEL_BRIDGE DRM_PANEL_EDP DRM_PANEL_ORIENTATION_QUIRKS DRM_SIMPLEDRM DRM_TTM DRM_TTM_HELPER DRM_UDL DRM_VGEM DRM_VIRTIO_GPU DRM_VIRTIO_GPU_KMS DRM_VKMS DRM_VMWGFX DUMMY DVB_AF9013 DVB_AF9033 DVB_AS102 DVB_AS102_FE DVB_B2C2_FLEXCOP DVB_B2C2_FLEXCOP_USB DVB_CORE DVB_DIB3000MB DVB_DIB3000MC DVB_EC100 DVB_GP8PSK_FE DVB_RTL2830 DVB_RTL2832 DVB_RTL2832_SDR DVB_TEST_DRIVERS DVB_TTUSB_BUDGET DVB_TTUSB_DEC DVB_USB DVB_USB_A800 DVB_USB_AF9005 DVB_USB_AF9005_REMOTE DVB_USB_AF9015 DVB_USB_AF9035 DVB_USB_ANYSEE DVB_USB_AU6610 DVB_USB_AZ6007 DVB_USB_AZ6027 DVB_USB_CE6230 DVB_USB_CINERGY_T2 DVB_USB_CXUSB DVB_USB_CXUSB_ANALOG DVB_USB_DIB0700 DVB_USB_DIB3000MC DVB_USB_DIBUSB_MB DVB_USB_DIBUSB_MC DVB_USB_DIGITV DVB_USB_DTT200U DVB_USB_DTV5100 DVB_USB_DVBSKY DVB_USB_DW2102 DVB_USB_EC168 DVB_USB_GL861 DVB_USB_GP8PSK DVB_USB_LME2510 DVB_USB_M920X DVB_USB_MXL111SF DVB_USB_NOVA_T_USB2 DVB_USB_OPERA1 DVB_USB_PCTV452E DVB_USB_RTL28XXU DVB_USB_TECHNISAT_USB2 DVB_USB_TTUSB2 DVB_USB_UMT_010 DVB_USB_V2 DVB_USB_VP702X DVB_USB_VP7045 DVB_USB_ZD1301 DVB_VIDTV DVB_ZL10353 ECRYPT_FS ECRYPT_FS_MESSAGING EDAC EEPROM_93CX6 EFS_FS ENCRYPTED_KEYS EQUALIZER EROFS_FS EROFS_FS_BACKED_BY_FILE EROFS_FS_POSIX_ACL EROFS_FS_SECURITY EROFS_FS_XATTR EROFS_FS_ZIP EVM EVM_ADD_XATTRS EVM_ATTR_FSUUID EXFAT_FS EXPORTFS_BLOCK_OPS EXT3_FS EXT3_FS_POSIX_ACL EXT3_FS_SECURITY EXTCON EXTCON_INTEL_CHT_WC EXTCON_PTN5150 EXTCON_USBC_TUSB320 F2FS_CHECK_FS F2FS_FAULT_INJECTION F2FS_FS F2FS_FS_COMPRESSION F2FS_FS_LZ4 F2FS_FS_LZ4HC F2FS_FS_LZO F2FS_FS_LZORLE F2FS_FS_POSIX_ACL F2FS_FS_SECURITY F2FS_FS_XATTR F2FS_FS_ZSTD F2FS_STAT_FS FANOTIFY FANOTIFY_ACCESS_PERMISSIONS FB FB_CFB_COPYAREA FB_CFB_FILLRECT FB_CFB_IMAGEBLIT FB_CORE FB_DEFERRED_IO FB_DEVICE FB_IOMEM_FOPS FB_IOMEM_HELPERS FB_NOTIFY FB_SYSMEM_FOPS FB_SYSMEM_HELPERS FB_SYSMEM_HELPERS_DEFERRED FB_SYS_COPYAREA FB_SYS_FILLRECT FB_SYS_IMAGEBLIT FB_TILEBLITTING FB_VESA FB_VGA16 FB_VIRTUAL FDDI FIREWIRE FIREWIRE_NET FIREWIRE_OHCI FIREWIRE_SBP2 FONT_8x16 FONT_8x8 FONT_SUPPORT FRAMEBUFFER_CONSOLE FRAMEBUFFER_CONSOLE_DETECT_PRIMARY FRAMEBUFFER_CONSOLE_ROTATION FSCACHE FS_DAX FS_DAX_PMD FS_ENCRYPTION FS_ENCRYPTION_ALGS FS_STACK FS_VERITY FS_VERITY_BUILTIN_SIGNATURES FTL FUSE_DAX FUSE_FS FW_LOADER_COMPRESS FW_LOADER_PAGED_BUF FW_LOADER_SYSFS FW_LOADER_USER_HELPER FW_LOADER_USER_HELPER_FALLBACK GACT_PROB GARP GENERIC_PHY GET_FREE_REGION GFS2_FS GFS2_FS_LOCKING_DLM GNSS GNSS_USB GOOGLE_COREBOOT_TABLE GOOGLE_FIRMWARE GOOGLE_MEMCONSOLE GOOGLE_MEMCONSOLE_COREBOOT GOOGLE_VPD GPIOLIB GPIOLIB_IRQCHIP GPIO_ACPI GPIO_DLN2 GPIO_LJCA GPIO_VIPERBOARD GREENASIA_FF GREYBUS GREYBUS_BRIDGED_PHY GREYBUS_ES2 GREYBUS_HID GREYBUS_USB GROUP_SCHED_WEIGHT GTP GUEST_PERF_EVENTS GVE HAMRADIO HAVE_ARCH_NODE_DEV_GROUP HAVE_ARCH_USERFAULTFD_MINOR HAVE_ARCH_USERFAULTFD_WP HAVE_BOOTMEM_INFO_NODE HAVE_CLK_PREPARE HAVE_KVM_CPU_RELAX_INTERCEPT HAVE_KVM_DIRTY_RING HAVE_KVM_DIRTY_RING_ACQ_REL HAVE_KVM_DIRTY_RING_TSO HAVE_KVM_IRQCHIP HAVE_KVM_IRQ_BYPASS HAVE_KVM_IRQ_ROUTING HAVE_KVM_MSI HAVE_KVM_NO_POLL HAVE_KVM_PFNCACHE HAVE_KVM_PM_NOTIFIER HAVE_KVM_READONLY_MEM HAVE_SCHED_AVG_IRQ HDLC HDLC_CISCO HDLC_FR HDLC_PPP HDLC_RAW HDLC_RAW_ETH HDLC_X25 HDMI HFSPLUS_FS HFS_FS HID_ACCUTOUCH HID_ACRUX HID_ACRUX_FF HID_ALPS HID_APPLEIR HID_ASUS HID_AUREAL HID_BATTERY_STRENGTH HID_BETOP_FF HID_BIGBEN_FF HID_CMEDIA HID_CORSAIR HID_COUGAR HID_CP2112 HID_CREATIVE_SB0540 HID_DRAGONRISE HID_ELAN HID_ELECOM HID_ELO HID_EMS_FF HID_EVISION HID_FT260 HID_GEMBIRD HID_GFRM HID_GLORIOUS HID_GOOGLE_STADIA_FF HID_GREENASIA HID_GT683R HID_HOLTEK HID_ICADE HID_JABRA HID_KEYTOUCH HID_KYE HID_LCPOWER HID_LED HID_LENOVO HID_LETSKETCH HID_LOGITECH_DJ HID_LOGITECH_HIDPP HID_MACALLY HID_MAGICMOUSE HID_MALTRON HID_MAYFLASH HID_MCP2200 HID_MCP2221 HID_MEGAWORLD_FF HID_MULTITOUCH HID_NINTENDO HID_NTI HID_ORTEK HID_PENMOUNT HID_PICOLCD HID_PICOLCD_BACKLIGHT HID_PICOLCD_CIR HID_PICOLCD_FB HID_PICOLCD_LCD HID_PICOLCD_LEDS HID_PLANTRONICS HID_PRIMAX HID_PRODIKEYS HID_PXRC HID_RAZER HID_RETRODE HID_RMI HID_ROCCAT HID_SAITEK HID_SEMITEK HID_SENSOR_ACCEL_3D HID_SENSOR_ALS HID_SENSOR_CUSTOM_INTEL_HINGE HID_SENSOR_CUSTOM_SENSOR HID_SENSOR_DEVICE_ROTATION HID_SENSOR_GYRO_3D HID_SENSOR_HUB HID_SENSOR_HUMIDITY HID_SENSOR_IIO_COMMON HID_SENSOR_IIO_TRIGGER HID_SENSOR_INCLINOMETER_3D HID_SENSOR_MAGNETOMETER_3D HID_SENSOR_PRESS HID_SENSOR_PROX HID_SENSOR_TEMP HID_SIGMAMICRO HID_SPEEDLINK HID_STEELSERIES HID_THINGM HID_TIVO HID_TOPRE HID_TWINHAN HID_U2FZERO HID_UCLOGIC HID_UDRAW_PS3 HID_VIEWSONIC HID_VIVALDI HID_VIVALDI_COMMON HID_VRC2 HID_WACOM HID_WALTOP HID_WIIMOTE HID_XIAOMI HID_XINMO HID_ZEROPLUS HID_ZYDACRON HMM_MIRROR HOLTEK_FF HOTPLUG_PCI_PCIE HPET_MMAP HPET_MMAP_DEFAULT HPFS_FS HSR I2C_ALGOBIT I2C_CHARDEV I2C_CP2615 I2C_DESIGNWARE_CORE I2C_DESIGNWARE_PLATFORM I2C_DIOLAN_U2C I2C_DLN2 I2C_HID_ACPI I2C_HID_CORE I2C_HID_OF I2C_LJCA I2C_MUX I2C_MUX_REG I2C_ROBOTFUZZ_OSIF I2C_SI4713 I2C_SLAVE I2C_SLAVE_EEPROM I2C_TINY_USB I2C_VIPERBOARD IEEE802154 IEEE802154_6LOWPAN IEEE802154_ATUSB IEEE802154_DRIVERS IEEE802154_HWSIM IEEE802154_NL802154_EXPERIMENTAL IEEE802154_SOCKET IFB IIO IIO_BUFFER IIO_KFIFO_BUF IIO_TRIGGER IIO_TRIGGERED_BUFFER IKCONFIG IKCONFIG_PROC IMA IMA_APPRAISE IMA_APPRAISE_MODSIG IMA_DEFAULT_HASH_SHA256 IMA_LSM_RULES IMA_MEASURE_ASYMMETRIC_KEYS IMA_NG_TEMPLATE IMA_QUEUE_EARLY_BOOT_KEYS IMA_READ_POLICY IMA_WRITE_POLICY INET6_ESPINTCP INET6_ESP_OFFLOAD INET6_IPCOMP INET6_TUNNEL INET6_XFRM_TUNNEL INET_AH INET_DCCP_DIAG INET_DIAG INET_DIAG_DESTROY INET_ESP INET_ESPINTCP INET_ESP_OFFLOAD INET_IPCOMP INET_MPTCP_DIAG INET_RAW_DIAG INET_SCTP_DIAG INET_TCP_DIAG INET_UDP_DIAG INET_XFRM_TUNNEL INFINIBAND INFINIBAND_ADDR_TRANS INFINIBAND_ADDR_TRANS_CONFIGFS INFINIBAND_IPOIB INFINIBAND_IPOIB_CM INFINIBAND_IPOIB_DEBUG INFINIBAND_ISER INFINIBAND_ON_DEMAND_PAGING INFINIBAND_RTRS INFINIBAND_RTRS_CLIENT INFINIBAND_SRP INFINIBAND_USER_ACCESS INFINIBAND_USER_MAD INFINIBAND_USER_MEM INPUT_ATI_REMOTE2 INPUT_CM109 INPUT_IMS_PCU INPUT_JOYDEV INPUT_JOYSTICK INPUT_KEYSPAN_REMOTE INPUT_LEDS INPUT_MOUSE INPUT_MOUSEDEV INPUT_MOUSEDEV_PSAUX INPUT_POWERMATE INPUT_UINPUT INPUT_YEALINK INTEGRITY INTEGRITY_ASYMMETRIC_KEYS INTEGRITY_AUDIT INTEGRITY_SIGNATURE INTEGRITY_TRUSTED_KEYRING INTEL_CHTWC_INT33FE INTEL_IDMA64 INTEL_IOATDMA INTEL_IOMMU_DEFAULT_ON INTEL_IOMMU_SVM INTEL_ISHTP_ECLITE INTEL_ISH_FIRMWARE_DOWNLOADER INTEL_ISH_HID INTEL_SOC_PMIC_CHTWC INTERVAL_TREE_SPAN_ITER IOMMUFD IOMMUFD_DRIVER IOMMUFD_DRIVER_CORE IOMMUFD_TEST IOSCHED_BFQ IP6_NF_MATCH_AH IP6_NF_MATCH_EUI64 IP6_NF_MATCH_FRAG IP6_NF_MATCH_HL IP6_NF_MATCH_MH IP6_NF_MATCH_OPTS IP6_NF_MATCH_RPFILTER IP6_NF_MATCH_RT IP6_NF_MATCH_SRH IP6_NF_NAT IP6_NF_RAW IP6_NF_SECURITY IP6_NF_TARGET_HL IP6_NF_TARGET_MASQUERADE IP6_NF_TARGET_NPT IP6_NF_TARGET_SYNPROXY IPV6_FOU IPV6_FOU_TUNNEL IPV6_GRE IPV6_ILA IPV6_MIP6 IPV6_MROUTE IPV6_MROUTE_MULTIPLE_TABLES IPV6_MULTIPLE_TABLES IPV6_OPTIMISTIC_DAD IPV6_PIMSM_V2 IPV6_ROUTER_PREF IPV6_ROUTE_INFO IPV6_RPL_LWTUNNEL IPV6_SEG6_BPF IPV6_SEG6_HMAC IPV6_SEG6_LWTUNNEL IPV6_SIT_6RD IPV6_SUBTREES IPV6_TUNNEL IPV6_VTI IPVLAN IPVLAN_L3S IPVTAP IP_DCCP IP_DCCP_CCID3 IP_DCCP_TFRC_LIB IP_FIB_TRIE_STATS IP_MROUTE_MULTIPLE_TABLES IP_NF_ARPFILTER IP_NF_ARPTABLES IP_NF_ARP_MANGLE IP_NF_MATCH_AH IP_NF_MATCH_ECN IP_NF_MATCH_RPFILTER IP_NF_MATCH_TTL IP_NF_RAW IP_NF_SECURITY IP_NF_TARGET_ECN IP_NF_TARGET_NETMAP IP_NF_TARGET_REDIRECT IP_NF_TARGET_SYNPROXY IP_NF_TARGET_TTL IP_ROUTE_CLASSID IP_SCTP IP_SET IP_SET_BITMAP_IP IP_SET_BITMAP_IPMAC IP_SET_BITMAP_PORT IP_SET_HASH_IP IP_SET_HASH_IPMAC IP_SET_HASH_IPMARK IP_SET_HASH_IPPORT IP_SET_HASH_IPPORTIP IP_SET_HASH_IPPORTNET IP_SET_HASH_MAC IP_SET_HASH_NET IP_SET_HASH_NETIFACE IP_SET_HASH_NETNET IP_SET_HASH_NETPORT IP_SET_HASH_NETPORTNET IP_SET_LIST_SET IP_VS IP_VS_DH IP_VS_FO IP_VS_FTP IP_VS_IPV6 IP_VS_LBLC IP_VS_LBLCR IP_VS_LC IP_VS_MH IP_VS_NFCT IP_VS_NQ IP_VS_OVF IP_VS_PE_SIP IP_VS_PROTO_AH IP_VS_PROTO_AH_ESP IP_VS_PROTO_ESP IP_VS_PROTO_SCTP IP_VS_PROTO_TCP IP_VS_PROTO_UDP IP_VS_RR IP_VS_SED IP_VS_SH IP_VS_TWOS IP_VS_WLC IP_VS_WRR IRQ_BYPASS_MANAGER IRQ_POLL IRQ_REMAP IRQ_TIME_ACCOUNTING IR_IGORPLUGUSB IR_IGUANA IR_IMON IR_IMON_RAW IR_MCEUSB IR_REDRAT3 IR_STREAMZAP IR_TOY IR_TTUSBIR ISDN ISDN_CAPI JFFS2_CMODE_PRIORITY JFFS2_COMPRESSION_OPTIONS JFFS2_FS JFFS2_FS_POSIX_ACL JFFS2_FS_SECURITY JFFS2_FS_WRITEBUFFER JFFS2_FS_XATTR JFFS2_LZO JFFS2_RTIME JFFS2_RUBIN JFFS2_SUMMARY JFFS2_ZLIB JFS_DEBUG JFS_FS JFS_POSIX_ACL JFS_SECURITY JOYSTICK_IFORCE JOYSTICK_IFORCE_USB JOYSTICK_PXRC JOYSTICK_XPAD JOYSTICK_XPAD_FF JOYSTICK_XPAD_LEDS KARMA_PARTITION KCOV KCOV_ENABLE_COMPARISONS KCOV_INSTRUMENT_ALL KEYS_REQUEST_CACHE KEY_DH_OPERATIONS KEY_NOTIFICATIONS KSM KVM KVM_AMD KVM_ASYNC_PF KVM_COMMON KVM_COMPAT KVM_ELIDE_TLB_FLUSH_IF_YOUNG KVM_GENERIC_DIRTYLOG_READ_PROTECT KVM_GENERIC_HARDWARE_ENABLING KVM_GENERIC_MEMORY_ATTRIBUTES KVM_GENERIC_MMU_NOTIFIER KVM_GENERIC_PRE_FAULT_MEMORY KVM_GENERIC_PRIVATE_MEM KVM_HYPERV KVM_INTEL KVM_MMIO KVM_PRIVATE_MEM KVM_PROVE_MMU KVM_SW_PROTECTED_VM KVM_VFIO KVM_X86 KVM_XEN KVM_XFER_TO_GUEST_WORK L2TP L2TP_ETH L2TP_IP L2TP_V3 LAPB LAPBETHER LCD_CLASS_DEVICE LDM_PARTITION LEGACY_PTYS LIBCRC32C LIBNVDIMM LINEAR_RANGES LLC LLC2 LOGIG940_FF LOGIRUMBLEPAD2_FF LOGO LOGO_LINUX_MONO LOGO_LINUX_VGA16 LPC_ICH LWTUNNEL LWTUNNEL_BPF LZ4HC_COMPRESS LZ4_COMPRESS MAC80211 MAC80211_DEBUGFS MAC80211_HAS_RC MAC80211_HWSIM MAC80211_LEDS MAC80211_MESH MAC80211_RC_DEFAULT_MINSTREL MAC80211_RC_MINSTREL MAC802154 MACSEC MACVLAN MACVTAP MAC_PARTITION MAPPING_DIRTY_HELPERS MDIO_MVUSB MD_RAID0 MD_RAID1 MD_RAID10 MD_RAID456 MEDIA_ANALOG_TV_SUPPORT MEDIA_ATTACH MEDIA_CAMERA_SUPPORT MEDIA_CEC_SUPPORT MEDIA_CONTROLLER MEDIA_CONTROLLER_DVB MEDIA_DIGITAL_TV_SUPPORT MEDIA_RADIO_SUPPORT MEDIA_SDR_SUPPORT MEDIA_SUPPORT MEDIA_SUPPORT_FILTER MEDIA_TEST_SUPPORT MEDIA_TUNER MEDIA_TUNER_MSI001 MEDIA_USB_SUPPORT MEMORY_BALLOON MEMORY_HOTPLUG MEMORY_HOTPLUG_DEFAULT_ONLINE MEMORY_HOTREMOVE MEMORY_ISOLATION MEMREGION MEMSTICK MEMSTICK_REALTEK_USB MEM_SOFT_DIRTY MFD_CORE MFD_DLN2 MFD_SYSCON MFD_VIPERBOARD MHI_BUS MHI_WWAN_CTRL MHP_MEMMAP_ON_MEMORY MICROCHIP_PHY MINIX_FS MINIX_SUBPARTITION MIN_HEAP MISC_RTSX MISC_RTSX_USB MISDN MISDN_DSP MISDN_HFCUSB MISDN_L1OIP MKISS MLX4_CORE MLX4_INFINIBAND MMC MMC_REALTEK_USB MMC_USHC MMC_VUB300 MODULE_SRCVERSION_ALL MODVERSIONS MOST MOST_USB_HDM MOUSE_APPLETOUCH MOUSE_BCM5974 MOUSE_PS2 MOUSE_PS2_ALPS MOUSE_PS2_BYD MOUSE_PS2_CYPRESS MOUSE_PS2_FOCALTECH MOUSE_PS2_LIFEBOOK MOUSE_PS2_LOGIPS2PP MOUSE_PS2_SMBUS MOUSE_PS2_SYNAPTICS MOUSE_PS2_SYNAPTICS_SMBUS MOUSE_PS2_TRACKPOINT MOUSE_SYNAPTICS_USB MPLS MPLS_IPTUNNEL MPLS_ROUTING MPTCP MPTCP_IPV6 MRP MTD MTD_BLKDEVS MTD_BLOCK MTD_BLOCK2MTD MTD_CFI_I1 MTD_CFI_I2 MTD_MAP_BANK_WIDTH_1 MTD_MAP_BANK_WIDTH_2 MTD_MAP_BANK_WIDTH_4 MTD_MTDRAM MTD_PHRAM MTD_SLRAM MUSB_PIO_ONLY ND_BTT ND_CLAIM ND_PFN NETDEVSIM NETFILTER_ADVANCED NETFILTER_BPF_LINK NETFILTER_FAMILY_ARP NETFILTER_FAMILY_BRIDGE NETFILTER_NETLINK_ACCT NETFILTER_NETLINK_GLUE_CT NETFILTER_NETLINK_OSF NETFILTER_NETLINK_QUEUE NETFILTER_SYNPROXY NETFILTER_XTABLES_COMPAT NETFILTER_XT_CONNMARK NETFILTER_XT_MATCH_BPF NETFILTER_XT_MATCH_CGROUP NETFILTER_XT_MATCH_CLUSTER NETFILTER_XT_MATCH_COMMENT NETFILTER_XT_MATCH_CONNBYTES NETFILTER_XT_MATCH_CONNLABEL NETFILTER_XT_MATCH_CONNLIMIT NETFILTER_XT_MATCH_CONNMARK NETFILTER_XT_MATCH_CPU NETFILTER_XT_MATCH_DCCP NETFILTER_XT_MATCH_DEVGROUP NETFILTER_XT_MATCH_DSCP NETFILTER_XT_MATCH_ECN NETFILTER_XT_MATCH_ESP NETFILTER_XT_MATCH_HASHLIMIT NETFILTER_XT_MATCH_HELPER NETFILTER_XT_MATCH_HL NETFILTER_XT_MATCH_IPCOMP NETFILTER_XT_MATCH_IPRANGE NETFILTER_XT_MATCH_IPVS NETFILTER_XT_MATCH_L2TP NETFILTER_XT_MATCH_LENGTH NETFILTER_XT_MATCH_LIMIT NETFILTER_XT_MATCH_MAC NETFILTER_XT_MATCH_MARK NETFILTER_XT_MATCH_MULTIPORT NETFILTER_XT_MATCH_NFACCT NETFILTER_XT_MATCH_OSF NETFILTER_XT_MATCH_OWNER NETFILTER_XT_MATCH_PHYSDEV NETFILTER_XT_MATCH_PKTTYPE NETFILTER_XT_MATCH_QUOTA NETFILTER_XT_MATCH_RATEEST NETFILTER_XT_MATCH_REALM NETFILTER_XT_MATCH_RECENT NETFILTER_XT_MATCH_SCTP NETFILTER_XT_MATCH_SOCKET NETFILTER_XT_MATCH_STATISTIC NETFILTER_XT_MATCH_STRING NETFILTER_XT_MATCH_TCPMSS NETFILTER_XT_MATCH_TIME NETFILTER_XT_MATCH_U32 NETFILTER_XT_SET NETFILTER_XT_TARGET_AUDIT NETFILTER_XT_TARGET_CHECKSUM NETFILTER_XT_TARGET_CLASSIFY NETFILTER_XT_TARGET_CONNMARK NETFILTER_XT_TARGET_CT NETFILTER_XT_TARGET_DSCP NETFILTER_XT_TARGET_HL NETFILTER_XT_TARGET_HMARK NETFILTER_XT_TARGET_IDLETIMER NETFILTER_XT_TARGET_LED NETFILTER_XT_TARGET_MARK NETFILTER_XT_TARGET_NETMAP NETFILTER_XT_TARGET_NFQUEUE NETFILTER_XT_TARGET_NOTRACK NETFILTER_XT_TARGET_RATEEST NETFILTER_XT_TARGET_REDIRECT NETFILTER_XT_TARGET_TCPOPTSTRIP NETFILTER_XT_TARGET_TEE NETFILTER_XT_TARGET_TPROXY NETFILTER_XT_TARGET_TRACE NETLABEL NETLINK_DIAG NETROM NET_9P_RDMA NET_ACT_BPF NET_ACT_CONNMARK NET_ACT_CSUM NET_ACT_CT NET_ACT_CTINFO NET_ACT_GACT NET_ACT_GATE NET_ACT_IFE NET_ACT_MIRRED NET_ACT_MPLS NET_ACT_NAT NET_ACT_PEDIT NET_ACT_POLICE NET_ACT_SAMPLE NET_ACT_SIMP NET_ACT_SKBEDIT NET_ACT_SKBMOD NET_ACT_TUNNEL_KEY NET_ACT_VLAN NET_CLS_BASIC NET_CLS_BPF NET_CLS_FLOW NET_CLS_FLOWER NET_CLS_FW NET_CLS_MATCHALL NET_CLS_ROUTE4 NET_CLS_U32 NET_DEVLINK NET_DEVMEM NET_DROP_MONITOR NET_DSA NET_DSA_TAG_BRCM NET_DSA_TAG_BRCM_COMMON NET_DSA_TAG_BRCM_PREPEND NET_DSA_TAG_MTK NET_DSA_TAG_QCA NET_DSA_TAG_RTL4_A NET_EMATCH_CANID NET_EMATCH_CMP NET_EMATCH_IPSET NET_EMATCH_IPT NET_EMATCH_META NET_EMATCH_NBYTE NET_EMATCH_TEXT NET_EMATCH_U32 NET_FC NET_FOU NET_FOU_IP_TUNNELS NET_IFE NET_IFE_SKBMARK NET_IFE_SKBPRIO NET_IFE_SKBTCINDEX NET_IPGRE NET_IPGRE_BROADCAST NET_IPGRE_DEMUX NET_IPIP NET_IPVTI NET_KEY NET_KEY_MIGRATE NET_L3_MASTER_DEV NET_MPLS_GSO NET_NCSI NET_NSH NET_REDIRECT NET_SCH_CAKE NET_SCH_CBS NET_SCH_CHOKE NET_SCH_CODEL NET_SCH_DEFAULT NET_SCH_DRR NET_SCH_ETF NET_SCH_ETS NET_SCH_FQ NET_SCH_FQ_CODEL NET_SCH_FQ_PIE NET_SCH_GRED NET_SCH_HFSC NET_SCH_HHF NET_SCH_HTB NET_SCH_INGRESS NET_SCH_MQPRIO NET_SCH_MQPRIO_LIB NET_SCH_MULTIQ NET_SCH_NETEM NET_SCH_PIE NET_SCH_PLUG NET_SCH_PRIO NET_SCH_QFQ NET_SCH_RED NET_SCH_SFB NET_SCH_SFQ NET_SCH_SKBPRIO NET_SCH_TAPRIO NET_SCH_TBF NET_SCH_TEQL NET_SHAPER NET_SOCK_MSG NET_SWITCHDEV NET_TC_SKB_EXT NET_TEAM NET_TEAM_MODE_ACTIVEBACKUP NET_TEAM_MODE_BROADCAST NET_TEAM_MODE_LOADBALANCE NET_TEAM_MODE_RANDOM NET_TEAM_MODE_ROUNDROBIN NET_UDP_TUNNEL NET_VENDOR_HISILICON NET_VRF NFC NFC_DIGITAL NFC_FDP NFC_HCI NFC_MRVL NFC_MRVL_USB NFC_NCI NFC_NCI_UART NFC_PN533 NFC_PN533_USB NFC_PORT100 NFC_SHDLC NFC_SIM NFC_VIRTUAL_NCI NFSD NFSD_BLOCKLAYOUT NFSD_FLEXFILELAYOUT NFSD_PNFS NFSD_SCSILAYOUT NFSD_V3_ACL NFSD_V4 NFSD_V4_2_INTER_SSC NFSD_V4_SECURITY_LABEL NFS_FSCACHE NFS_V4_1 NFS_V4_2 NFS_V4_2_READ_PLUS NFS_V4_2_SSC_HELPER NFS_V4_SECURITY_LABEL NFT_BRIDGE_META NFT_BRIDGE_REJECT NFT_COMPAT NFT_COMPAT_ARP NFT_CONNLIMIT NFT_CT NFT_DUP_IPV4 NFT_DUP_IPV6 NFT_DUP_NETDEV NFT_FIB NFT_FIB_INET NFT_FIB_IPV4 NFT_FIB_IPV6 NFT_FIB_NETDEV NFT_FLOW_OFFLOAD NFT_FWD_NETDEV NFT_HASH NFT_LIMIT NFT_LOG NFT_MASQ NFT_NAT NFT_NUMGEN NFT_OSF NFT_QUEUE NFT_QUOTA NFT_REDIR NFT_REJECT NFT_REJECT_INET NFT_REJECT_IPV4 NFT_REJECT_IPV6 NFT_REJECT_NETDEV NFT_SOCKET NFT_SYNPROXY NFT_TPROXY NFT_TUNNEL NFT_XFRM NF_CONNTRACK_AMANDA NF_CONNTRACK_BRIDGE NF_CONNTRACK_BROADCAST NF_CONNTRACK_EVENTS NF_CONNTRACK_H323 NF_CONNTRACK_LABELS NF_CONNTRACK_MARK NF_CONNTRACK_NETBIOS_NS NF_CONNTRACK_OVS NF_CONNTRACK_PPTP NF_CONNTRACK_SANE NF_CONNTRACK_SNMP NF_CONNTRACK_TFTP NF_CONNTRACK_TIMEOUT NF_CONNTRACK_TIMESTAMP NF_CONNTRACK_ZONES NF_CT_NETLINK_HELPER NF_CT_NETLINK_TIMEOUT NF_CT_PROTO_DCCP NF_CT_PROTO_GRE NF_CT_PROTO_SCTP NF_CT_PROTO_UDPLITE NF_DUP_IPV4 NF_DUP_IPV6 NF_DUP_NETDEV NF_FLOW_TABLE NF_FLOW_TABLE_INET NF_NAT_AMANDA NF_NAT_H323 NF_NAT_OVS NF_NAT_PPTP NF_NAT_REDIRECT NF_NAT_SNMP_BASIC NF_NAT_TFTP NF_SOCKET_IPV4 NF_SOCKET_IPV6 NF_TABLES NF_TABLES_ARP NF_TABLES_BRIDGE NF_TABLES_INET NF_TABLES_IPV4 NF_TABLES_IPV6 NF_TABLES_NETDEV NF_TPROXY_IPV4 NF_TPROXY_IPV6 NILFS2_FS NINTENDO_FF NLMON NLS_CODEPAGE_1250 NLS_CODEPAGE_1251 NLS_CODEPAGE_737 NOP_USB_XCEIV PAGE_POOL PARPORT PARTITION_ADVANCED PCCARD PCMCIA PHONET PSAMPLE RADIO_ADAPTERS RADIO_SI470X RADIO_SI4713 RAS RC_CORE RC_DEVICES REGULATOR RFKILL SERIAL_DEV_BUS SND SOUND SPI SSB STAGING TAP TARGET_CORE TLS TLS_DEVICE TRANSPARENT_HUGEPAGE TRUSTED_KEYS TUN TYPEC TYPEC_MUX_PI3USB30532 USB_AMD5536UDC USB_ATM USB_CDNS3 USB_CDNS3_HOST USB_CDNS3_PCI_WRAP USB_CDNSP_GADGET USB_CDNSP_HOST USB_CDNSP_PCI USB_CDNS_HOST USB_CDNS_SUPPORT USB_CHAOSKEY USB_CHIPIDEA USB_CHIPIDEA_GENERIC USB_CHIPIDEA_HOST USB_CHIPIDEA_MSM USB_CHIPIDEA_NPCM USB_CHIPIDEA_PCI USB_CHIPIDEA_UDC USB_CONFIGFS USB_CONFIGFS_ACM USB_CONFIGFS_ECM USB_CONFIGFS_ECM_SUBSET USB_CONFIGFS_EEM USB_CONFIGFS_F_FS USB_CONFIGFS_F_HID USB_CONFIGFS_F_LB_SS USB_CONFIGFS_F_MIDI USB_CONFIGFS_F_MIDI2 USB_CONFIGFS_F_PRINTER USB_CONFIGFS_F_TCM USB_CONFIGFS_F_UAC1 USB_CONFIGFS_F_UAC1_LEGACY USB_CONFIGFS_F_UAC2 USB_CONFIGFS_F_UVC USB_CONFIGFS_MASS_STORAGE USB_CONFIGFS_NCM USB_CONFIGFS_OBEX USB_CONFIGFS_PHONET USB_CONFIGFS_RNDIS USB_CONFIGFS_SERIAL USB_CONN_GPIO USB_CXACRU USB_CYPRESS_CY7C63 USB_CYTHERM USB_DSBR USB_DUMMY_HCD USB_DWC2 USB_DWC2_HOST USB_DWC2_PCI USB_DWC3 USB_DWC3_GADGET USB_DWC3_HAPS USB_DWC3_OF_SIMPLE USB_DWC3_PCI USB_DWC3_ULPI USB_DYNAMIC_MINORS USB_EG20T USB_EHCI_FSL USB_EHCI_HCD_PLATFORM USB_EHCI_ROOT_HUB_TT USB_EHSET_TEST_FIXTURE USB_EMI26 USB_EMI62 USB_EPSON2888 USB_EZUSB_FX2 USB_FEW_INIT_RETRIES USB_F_ACM USB_F_ECM USB_F_EEM USB_F_FS USB_F_HID USB_F_MASS_STORAGE USB_F_MIDI USB_F_MIDI2 USB_F_NCM USB_F_OBEX USB_F_PHONET USB_F_PRINTER USB_F_RNDIS USB_F_SERIAL USB_F_SS_LB USB_F_SUBSET USB_F_TCM USB_F_UAC1 USB_F_UAC1_LEGACY USB_F_UAC2 USB_F_UVC USB_GADGET USB_GADGETFS USB_GADGET_DEBUG_FILES USB_GADGET_DEBUG_FS USB_GL860 USB_GOKU USB_GR_UDC USB_GSPCA USB_GSPCA_BENQ USB_GSPCA_CONEX USB_GSPCA_CPIA1 USB_GSPCA_DTCS033 USB_GSPCA_ETOMS USB_GSPCA_FINEPIX USB_GSPCA_JEILINJ USB_GSPCA_JL2005BCD USB_GSPCA_KINECT USB_GSPCA_KONICA USB_GSPCA_MARS USB_GSPCA_MR97310A USB_GSPCA_NW80X USB_GSPCA_OV519 USB_GSPCA_OV534 USB_GSPCA_OV534_9 USB_GSPCA_PAC207 USB_GSPCA_PAC7302 USB_GSPCA_PAC7311 USB_GSPCA_SE401 USB_GSPCA_SN9C2028 USB_GSPCA_SN9C20X USB_GSPCA_SONIXB USB_GSPCA_SONIXJ USB_GSPCA_SPCA1528 USB_GSPCA_SPCA500 USB_GSPCA_SPCA501 USB_GSPCA_SPCA505 USB_GSPCA_SPCA506 USB_GSPCA_SPCA508 USB_GSPCA_SPCA561 USB_GSPCA_SQ905 USB_GSPCA_SQ905C USB_GSPCA_SQ930X USB_GSPCA_STK014 USB_GSPCA_STK1135 USB_GSPCA_STV0680 USB_GSPCA_SUNPLUS USB_GSPCA_T613 USB_GSPCA_TOPRO USB_GSPCA_TOUPTEK USB_GSPCA_TV8532 USB_GSPCA_VC032X USB_GSPCA_VICAM USB_GSPCA_XIRLINK_CIT USB_GSPCA_ZC3XX USB_HACKRF USB_HCD_BCMA USB_HCD_SSB USB_HSIC_USB3503 USB_HSIC_USB4604 USB_HSO USB_HUB_USB251XB USB_IDMOUSE USB_IOWARRIOR USB_IPHETH USB_ISIGHTFW USB_ISP116X_HCD USB_ISP1301 USB_ISP1760 USB_ISP1760_DUAL_ROLE USB_ISP1760_HCD USB_ISP1761_UDC USB_KAWETH USB_KC2190 USB_KEENE USB_LAN78XX USB_LCD USB_LD USB_LEDS_TRIGGER_USBPORT USB_LED_TRIG USB_LEGOTOWER USB_LGM_PHY USB_LIBCOMPOSITE USB_LINK_LAYER_TEST USB_LJCA USB_M5602 USB_MA901 USB_MAX3420_UDC USB_MAX3421_HCD USB_MDC800 USB_MICROTEK USB_MR800 USB_MSI2500 USB_MUSB_DUAL_ROLE USB_MUSB_HDRC USB_MV_U3D USB_MV_UDC USB_NET2272 USB_NET2272_DMA USB_NET2280 USB_NET_AQC111 USB_NET_AX88179_178A USB_NET_AX8817X USB_NET_CDCETHER USB_NET_CDC_EEM USB_NET_CDC_MBIM USB_NET_CDC_NCM USB_NET_CDC_SUBSET USB_NET_CDC_SUBSET_ENABLE USB_NET_CH9200 USB_NET_CX82310_ETH USB_NET_DM9601 USB_NET_GL620A USB_NET_HUAWEI_CDC_NCM USB_NET_INT51X1 USB_NET_KALMIA USB_NET_MCS7830 USB_NET_NET1080 USB_NET_PLUSB USB_NET_QMI_WWAN USB_NET_RNDIS_HOST USB_NET_SMSC75XX USB_NET_SMSC95XX USB_NET_SR9700 USB_NET_SR9800 USB_NET_ZAURUS USB_OHCI_HCD_PLATFORM USB_OTG USB_OTG_FSM USB_OXU210HP_HCD USB_PEGASUS USB_PHY USB_PULSE8_CEC USB_PWC USB_PWC_INPUT_EVDEV USB_PXA27X USB_R8A66597 USB_R8A66597_HCD USB_RAINSHADOW_CEC USB_RAREMONO USB_RAW_GADGET USB_ROLES_INTEL_XHCI USB_ROLE_SWITCH USB_RTL8150 USB_RTL8152 USB_RTL8153_ECM USB_S2255 USB_SERIAL USB_SERIAL_AIRCABLE USB_SERIAL_ARK3116 USB_SERIAL_BELKIN USB_SERIAL_CH341 USB_SERIAL_CONSOLE USB_SERIAL_CP210X USB_SERIAL_CYBERJACK USB_SERIAL_CYPRESS_M8 USB_SERIAL_DEBUG USB_SERIAL_DIGI_ACCELEPORT USB_SERIAL_EDGEPORT USB_SERIAL_EDGEPORT_TI USB_SERIAL_EMPEG USB_SERIAL_F81232 USB_SERIAL_F8153X USB_SERIAL_FTDI_SIO USB_SERIAL_GARMIN USB_SERIAL_GENERIC USB_SERIAL_IPAQ USB_SERIAL_IPW USB_SERIAL_IR USB_SERIAL_IUU USB_SERIAL_KEYSPAN USB_SERIAL_KEYSPAN_PDA USB_SERIAL_KLSI USB_SERIAL_KOBIL_SCT USB_SERIAL_MCT_U232 USB_SERIAL_METRO USB_SERIAL_MOS7715_PARPORT USB_SERIAL_MOS7720 USB_SERIAL_MOS7840 USB_SERIAL_MXUPORT USB_SERIAL_NAVMAN USB_SERIAL_OMNINET USB_SERIAL_OPTICON USB_SERIAL_OPTION USB_SERIAL_OTI6858 USB_SERIAL_PL2303 USB_SERIAL_QCAUX USB_SERIAL_QT2 USB_SERIAL_QUALCOMM USB_SERIAL_SAFE USB_SERIAL_SIERRAWIRELESS USB_SERIAL_SIMPLE USB_SERIAL_SPCP8X5 USB_SERIAL_SSU100 USB_SERIAL_SYMBOL USB_SERIAL_TI USB_SERIAL_UPD78F0730 USB_SERIAL_VISOR USB_SERIAL_WHITEHEAT USB_SERIAL_WISHBONE USB_SERIAL_WWAN USB_SERIAL_XR USB_SERIAL_XSENS_MT USB_SEVSEG USB_SI470X USB_SI4713 USB_SIERRA_NET USB_SISUSBVGA USB_SL811_CS USB_SL811_HCD USB_SL811_HCD_ISO USB_SNP_CORE USB_SPEEDTOUCH USB_STORAGE_ALAUDA USB_STORAGE_CYPRESS_ATACB USB_STORAGE_DATAFAB USB_STORAGE_ENE_UB6250 USB_STORAGE_FREECOM USB_STORAGE_ISD200 USB_STORAGE_JUMPSHOT USB_STORAGE_KARMA USB_STORAGE_ONETOUCH USB_STORAGE_SDDR09 USB_STORAGE_SDDR55 USB_STORAGE_USBAT USB_STV06XX USB_TEST USB_TMC USB_TRANCEVIBRATOR USB_UAS USB_UEAGLEATM USB_ULPI_BUS USB_USBNET USB_USS720 USB_U_AUDIO USB_U_ETHER USB_U_SERIAL USB_VIDEO_CLASS USB_VIDEO_CLASS_INPUT_EVDEV USB_VL600 USB_WDM USB_XHCI_DBGCAP USB_XHCI_PCI_RENESAS USB_XHCI_PLATFORM USB_XUSBATM USB_YUREX USERFAULTFD USERIO USERMODE_DRIVER USER_RETURN_NOTIFIER UVC_COMMON U_SERIAL_CONSOLE V4L2_MEM2MEM_DEV V4L_TEST_DRIVERS VALIDATE_FS_PARSER VDPA VDPA_SIM VDPA_SIM_BLOCK VDPA_SIM_NET VETH VFIO VFIO_DEVICE_CDEV VFIO_PCI VFIO_PCI_CORE VFIO_PCI_INTX VFIO_PCI_MMAP VFIO_VIRQFD VGASTATE VHOST VHOST_CROSS_ENDIAN_LEGACY VHOST_IOTLB VHOST_NET VHOST_RING VHOST_TASK VHOST_VDPA VHOST_VSOCK VIDEO VIDEOBUF2_CORE VIDEOBUF2_DMA_CONTIG VIDEOBUF2_DMA_SG VIDEOBUF2_MEMOPS VIDEOBUF2_V4L2 VIDEOBUF2_VMALLOC VIDEOMODE_HELPERS VIDEO_AU0828 VIDEO_AU0828_RC VIDEO_AU0828_V4L2 VIDEO_CS53L32A VIDEO_CX231XX VIDEO_CX231XX_ALSA VIDEO_CX231XX_DVB VIDEO_CX231XX_RC VIDEO_CX2341X VIDEO_CX25840 VIDEO_DEV VIDEO_EM28XX VIDEO_EM28XX_ALSA VIDEO_EM28XX_DVB VIDEO_EM28XX_RC VIDEO_EM28XX_V4L2 VIDEO_GO7007 VIDEO_GO7007_LOADER VIDEO_GO7007_USB VIDEO_GO7007_USB_S2250_BOARD VIDEO_HDPVR VIDEO_MSP3400 VIDEO_PVRUSB2 VIDEO_PVRUSB2_DVB VIDEO_PVRUSB2_SYSFS VIDEO_SAA711X VIDEO_STK1160 VIDEO_TUNER VIDEO_TVEEPROM VIDEO_USBTV VIDEO_V4L2_I2C VIDEO_V4L2_SUBDEV_API VIDEO_V4L2_TPG VIDEO_VICODEC VIDEO_VIM2M VIDEO_VIMC VIDEO_VIVID VIDEO_VIVID_CEC VIDEO_WM8775 VIPERBOARD_ADC VIRTIO_BALLOON VIRTIO_DMA_SHARED_BUFFER VIRTIO_FS VIRTIO_MEM VIRTIO_MMIO VIRTIO_MMIO_CMDLINE_DEVICES VIRTIO_PMEM VIRTIO_VDPA VIRTIO_VSOCKETS VIRTIO_VSOCKETS_COMMON VIRT_WIFI VLAN_8021Q VLAN_8021Q_GVRP VLAN_8021Q_MVRP VMAP_PFN VMWARE_VMCI VMXNET3 VP_VDPA VSOCKETS VSOCKETS_DIAG VSOCKETS_LOOPBACK VSOCKMON VT_HW_CONSOLE_BINDING VXFS_FS WAN WANT_COMPAT_NETLINK_MESSAGES WANT_DEV_COREDUMP WATCH_QUEUE WEXT_CORE WEXT_PROC WIREGUARD WIRELESS WLAN WLAN_VENDOR_ADMTEK WLAN_VENDOR_ATH WLAN_VENDOR_SILABS WWAN X25 X86_SGX X86_SGX_KVM X86_USER_SHADOW_STACK X86_X2APIC X86_X32_ABI XDP_SOCKETS XDP_SOCKETS_DIAG XFRM_ESPINTCP XFRM_INTERFACE XFRM_IPCOMP XFRM_MIGRATE XFRM_OFFLOAD XFRM_STATISTICS XFRM_SUB_POLICY XFRM_USER_COMPAT XFS_FS XFS_POSIX_ACL XFS_QUOTA XFS_RT XILLYBUS_CLASS XILLYUSB XOR_BLOCKS YENTA YENTA_ENE_TUNE YENTA_O2 YENTA_RICOH YENTA_TI YENTA_TOSHIBA Z3FOLD Z3FOLD_DEPRECATED ZEROPLUS_FF ZLIB_DEFLATE ZONEFS_FS ZONE_DEVICE ZPOOL ZRAM ZRAM_BACKEND_FORCE_LZO ZRAM_BACKEND_LZO ZRAM_DEF_COMP_LZO ZSMALLOC ZSTD_COMPRESS ZSWAP ZSWAP_COMPRESSOR_DEFAULT_842 ZSWAP_DEFAULT_ON ZSWAP_SHRINKER_DEFAULT_ON ZSWAP_ZPOOL_DEFAULT_Z3FOLD_DEPRECATED] disabling configs for [UBSAN BUG KASAN LOCKDEP ATOMIC_SLEEP LEAK], they are not needed picked [v6.12 v6.11 v6.10 v6.8 v6.6 v6.4 v6.2 v6.0 v5.17 v5.14 v5.11 v5.8 v5.5 v5.2 v4.20 v4.19] out of 35 release tags testing release v6.12 testing commit adc218676eef25575469234709c2d87185ca223a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: f8126558f1355010f49e2c2be0690d177a9c7734274238f10edca8bcf76c26db run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: task hung in new_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: no output from test machine run #9: crashed: no output from test machine representative crash: INFO: rcu detected stall in corrupted, types: [HANG] testing release v6.11 testing commit 98f7e32f20d28ec452afb208f9cffc08448a2652 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 83f54d0581f9ef1120b6534a58fc4cbcb4cb02736386a0c4da31b245e6e095de run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: INFO: task hung in nsim_destroy run #8: crashed: no output from test machine run #9: OK representative crash: INFO: task hung in del_device_store, types: [HANG] testing release v6.10 testing commit 0c3836482481200ead7b416ca80c68a29cfdaabd gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 7b55b7153655319b6a71e3154917b8b6db25bb20c5105a378cdbce340f742cb4 run #0: crashed: INFO: task hung in new_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in new_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in new_device_store run #6: crashed: INFO: task hung in rtnetlink_rcv_msg run #7: crashed: no output from test machine run #8: OK run #9: crashed: no output from test machine representative crash: INFO: task hung in new_device_store, types: [HANG] testing release v6.8 testing commit e8f897f4afef0031fe618a8e94127a0934896aba gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3e72f7efa06a3de687d5ea5e093de1899b91b2d9f255318df1b3f8c2c2e4abaa run #0: crashed: INFO: task hung in switchdev_deferred_process_work run #1: crashed: INFO: task hung in devlink_pernet_pre_exit run #2: crashed: INFO: task hung in devlink_pernet_pre_exit run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in new_device_store run #6: crashed: INFO: task hung in new_device_store run #7: crashed: INFO: task hung in del_device_store run #8: crashed: no output from test machine run #9: crashed: INFO: task hung in addrconf_dad_work representative crash: INFO: task hung in switchdev_deferred_process_work, types: [HANG] testing release v6.6 testing commit ffc253263a1375a65fa6c9f62a893e9767fbebfa gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 19d55dc8d2bdeaed9aacd90de23e31509de119aebde60240aaebba6ef42843fe run #0: ignore: lost connection to test machine run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in new_device_store run #5: crashed: INFO: task hung in devlink_pernet_pre_exit run #6: crashed: INFO: task hung in devlink_pernet_pre_exit run #7: crashed: INFO: task hung in addrconf_dad_work run #8: crashed: INFO: task hung in addrconf_dad_work run #9: OK representative crash: INFO: task hung in del_device_store, types: [HANG] testing release v6.4 testing commit 6995e2de6891c724bfeb2db33d7b87775f913ad1 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: e6c030d2ca34a033ef97358a41a0b781c6898379855208cce21390bca5fd691f all runs: OK false negative chance: 0.000 # git bisect start ffc253263a1375a65fa6c9f62a893e9767fbebfa 6995e2de6891c724bfeb2db33d7b87775f913ad1 Bisecting: 14974 revisions left to test after this (roughly 14 steps) [692f5510159c79bfa312a4e27a15e266232bfb4c] Merge tag 'asoc-v6.6' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus testing commit 692f5510159c79bfa312a4e27a15e266232bfb4c gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: e70f2a3285f5085297a6ddba8080a557fd32b5ef739e1dc25dba27b343597141 run #0: crashed: INFO: task hung in devlink_pernet_pre_exit run #1: crashed: INFO: task hung in nsim_drv_remove run #2: crashed: INFO: task hung in devlink_pernet_pre_exit run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in devlink_pernet_pre_exit run #5: crashed: INFO: task hung in new_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: no output from test machine run #8: crashed: no output from test machine run #9: crashed: no output from test machine representative crash: INFO: task hung in devlink_pernet_pre_exit, types: [HANG UNKNOWN] # git bisect bad 692f5510159c79bfa312a4e27a15e266232bfb4c Bisecting: 7678 revisions left to test after this (roughly 13 steps) [b775d6c5859affe00527cbe74263de05cfe6b9f9] Merge tag 'mips_6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux testing commit b775d6c5859affe00527cbe74263de05cfe6b9f9 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 315281f20d2da7d8a00fabfbfa7209d53388985a5819e25e52b7b4c3edc0d916 run #0: crashed: INFO: task hung in devlink_pernet_pre_exit run #1: crashed: INFO: task hung in devlink_pernet_pre_exit run #2: crashed: INFO: task hung in devlink_pernet_pre_exit run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in devlink_pernet_pre_exit run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in del_device_store run #7: crashed: INFO: task hung in reg_check_chans_work run #8: crashed: no output from test machine run #9: crashed: no output from test machine representative crash: INFO: task hung in devlink_pernet_pre_exit, types: [HANG] # git bisect bad b775d6c5859affe00527cbe74263de05cfe6b9f9 Bisecting: 3061 revisions left to test after this (roughly 12 steps) [3a8a670eeeaa40d87bd38a587438952741980c18] Merge tag 'net-next-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next testing commit 3a8a670eeeaa40d87bd38a587438952741980c18 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9432770cbaf31979dd83d215efc7ef9992f950359dddf8a48dbd45cb1f61f5e3 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in devlink_pernet_pre_exit run #2: crashed: INFO: task hung in devlink_pernet_pre_exit run #3: crashed: INFO: task hung in new_device_store run #4: crashed: INFO: task hung in devlink_pernet_pre_exit run #5: crashed: INFO: task hung in nsim_drv_remove run #6: crashed: INFO: task hung in del_device_store run #7: crashed: no output from test machine run #8: crashed: no output from test machine run #9: OK representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad 3a8a670eeeaa40d87bd38a587438952741980c18 Bisecting: 2103 revisions left to test after this (roughly 11 steps) [6e17c6de3ddf3073741d9c91a796ee696914d8a0] Merge tag 'mm-stable-2023-06-24-19-15' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm testing commit 6e17c6de3ddf3073741d9c91a796ee696914d8a0 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a80aaa8c13c218142327f31aedeb908bd64afb61cc24b8cce93d5e4dd45105f8 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in new_device_store run #3: crashed: INFO: task hung in devlink_pernet_pre_exit run #4: crashed: INFO: task hung in devlink_pernet_pre_exit run #5: crashed: INFO: task hung in devlink_pernet_pre_exit run #6: crashed: INFO: task hung in crda_timeout_work run #7: crashed: no output from test machine run #8: OK run #9: crashed: no output from test machine representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad 6e17c6de3ddf3073741d9c91a796ee696914d8a0 Bisecting: 1048 revisions left to test after this (roughly 10 steps) [2605e80d3438c77190f55b821c6575048c68268e] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux testing commit 2605e80d3438c77190f55b821c6575048c68268e gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a47b1f4f90b2ff5b587abdb27754ef0b76b473cbf0da8c9fc70e96811d24ae4c run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in devlink_pernet_pre_exit run #2: crashed: INFO: task hung in nsim_drv_remove run #3: crashed: INFO: task hung in devlink_pernet_pre_exit run #4: crashed: INFO: task hung in devlink_pernet_pre_exit run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in devlink_pernet_pre_exit run #7: crashed: no output from test machine run #8: crashed: no output from test machine run #9: OK representative crash: INFO: task hung in del_device_store, types: [HANG] # git bisect bad 2605e80d3438c77190f55b821c6575048c68268e Bisecting: 665 revisions left to test after this (roughly 9 steps) [0aa69d53ac7c30f6184f88f2e310d808b32b35a5] Merge tag 'for-6.5/io_uring-2023-06-23' of git://git.kernel.dk/linux testing commit 0aa69d53ac7c30f6184f88f2e310d808b32b35a5 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 260043384fc3ef79a150f6d429748e9208620eebbec6296adc962b7645d65af4 all runs: OK false negative chance: 0.000 # git bisect good 0aa69d53ac7c30f6184f88f2e310d808b32b35a5 Bisecting: 341 revisions left to test after this (roughly 8 steps) [7cffdbe3607a6cc2dc02d135e13732ec36bc4e28] Merge tag 'x86-boot-2023-06-26' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 7cffdbe3607a6cc2dc02d135e13732ec36bc4e28 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3780520156b0a26315dea93d8f2f3643619a27192a94b472d20d186eb9ac8bbc run #0: crashed: INFO: task hung in nsim_drv_remove run #1: crashed: INFO: task hung in nsim_drv_remove run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in new_device_store run #4: crashed: INFO: task hung in devlink_pernet_pre_exit run #5: crashed: INFO: task hung in devlink_pernet_pre_exit run #6: crashed: INFO: task hung in devlink_pernet_pre_exit run #7: crashed: INFO: task hung in devlink_pernet_pre_exit run #8: crashed: INFO: task hung in nsim_drv_remove run #9: crashed: no output from test machine representative crash: INFO: task hung in nsim_drv_remove, types: [HANG] # git bisect bad 7cffdbe3607a6cc2dc02d135e13732ec36bc4e28 Bisecting: 161 revisions left to test after this (roughly 7 steps) [658afed19ceed54a52b9e9e69c0791c8868ff55d] mtd: block: use a simple bool to track open for write testing commit 658afed19ceed54a52b9e9e69c0791c8868ff55d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8a407edc508cf6a273680754fc7c75f13233f3180f40fa7c6cae4c88575846e2 all runs: OK false negative chance: 0.000 # git bisect good 658afed19ceed54a52b9e9e69c0791c8868ff55d Bisecting: 80 revisions left to test after this (roughly 6 steps) [0b24be4691c9e6ea13ca70050d42a9f9032fa788] splice: don't call file_accessed in copy_splice_read testing commit 0b24be4691c9e6ea13ca70050d42a9f9032fa788 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2cd20b4934c7a2d625d5db8f4cde4efca370486ff0d57b5505533be8e6d3acf7 all runs: OK false negative chance: 0.000 # git bisect good 0b24be4691c9e6ea13ca70050d42a9f9032fa788 Bisecting: 48 revisions left to test after this (roughly 5 steps) [f121ab7f4ac32ed2aa51035534926f9507a8308b] Merge tag 'irqchip-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms into irq/core testing commit f121ab7f4ac32ed2aa51035534926f9507a8308b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 7138f0ab87d68a578f8c1ef7c6fad30127d71493a8718827ccd4dd86cfb1fa13 run #0: crashed: INFO: task hung in new_device_store run #1: crashed: INFO: task hung in new_device_store run #2: crashed: INFO: task hung in devlink_pernet_pre_exit run #3: crashed: INFO: task hung in devlink_pernet_pre_exit run #4: crashed: INFO: task hung in nsim_drv_remove run #5: crashed: INFO: task hung in del_device_store run #6: crashed: INFO: task hung in rtnetlink_rcv_msg run #7: crashed: INFO: task hung in crda_timeout_work run #8: crashed: INFO: task hung in rtnetlink_rcv_msg run #9: crashed: no output from test machine representative crash: INFO: task hung in new_device_store, types: [HANG] # git bisect bad f121ab7f4ac32ed2aa51035534926f9507a8308b Bisecting: 15 revisions left to test after this (roughly 4 steps) [e0b78e9fa34102ce82ef03c71eb13f0b79954ba3] Merge branch irq/loongarch-fixes-6.5 into irq/irqchip-next testing commit e0b78e9fa34102ce82ef03c71eb13f0b79954ba3 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: f74d94550214cc09d87ce2542e602f0fe9a26e1e0ca2d7dda4eb4e8af01ae697 all runs: OK false negative chance: 0.000 # git bisect good e0b78e9fa34102ce82ef03c71eb13f0b79954ba3 Bisecting: 7 revisions left to test after this (roughly 3 steps) [48f31e496488a25f443c0df52464da446fb1d10c] irqchip/stm32-exti: Fix warning on initialized field overwritten testing commit 48f31e496488a25f443c0df52464da446fb1d10c gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 13c46c9b5c7ab0f716ce01dd1d73cc5b4950ae4114c197f8b6c5123887331259 all runs: OK false negative chance: 0.000 # git bisect good 48f31e496488a25f443c0df52464da446fb1d10c Bisecting: 3 revisions left to test after this (roughly 2 steps) [721255b9826bd11c7a38b585905fc2dd0fb94e52] genirq: Use a maple tree for interrupt descriptor management testing commit 721255b9826bd11c7a38b585905fc2dd0fb94e52 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 1344ea6c19682812c20d3a3a035c0f2a52f1984ed942d9430b408218cc900398 run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in devlink_free run #2: crashed: INFO: task hung in devlink_free run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in devlink_free run #5: crashed: INFO: task hung in devlink_pernet_pre_exit run #6: crashed: no output from test machine run #7: crashed: no output from test machine run #8: crashed: no output from test machine run #9: crashed: INFO: task hung in rtnl_newlink representative crash: INFO: task hung in del_device_store, types: [HANG UNKNOWN] # git bisect bad 721255b9826bd11c7a38b585905fc2dd0fb94e52 Bisecting: 1 revision left to test after this (roughly 1 step) [bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db] genirq: Use hlist for managing resend handlers testing commit bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: fdff4625daf12acfc6f8384249cb2be654f495b60fde53ea7dfbaa066dc4ce4d run #0: crashed: INFO: task hung in del_device_store run #1: crashed: INFO: task hung in del_device_store run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in devlink_free run #5: crashed: INFO: task hung in __tun_chr_ioctl run #6: crashed: no output from test machine run #7: crashed: no output from test machine run #8: crashed: no output from test machine run #9: crashed: INFO: task hung in devlink_pernet_pre_exit representative crash: INFO: task hung in del_device_store, types: [HANG UNKNOWN] # git bisect bad bc06a9e0874239cb6d4eebcb0ecd1a91ad9272db Bisecting: 0 revisions left to test after this (roughly 0 steps) [d15121be7485655129101f3960ae6add40204463] Revert "softirq: Let ksoftirqd do its job" testing commit d15121be7485655129101f3960ae6add40204463 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: abc76c232677e537ee7dc088a1e92896b5208436e2fd5009184425a3168474e7 run #0: crashed: INFO: task hung in devlink_pernet_pre_exit run #1: crashed: INFO: task hung in devlink_pernet_pre_exit run #2: crashed: INFO: task hung in del_device_store run #3: crashed: INFO: task hung in del_device_store run #4: crashed: INFO: task hung in del_device_store run #5: crashed: INFO: task hung in devlink_pernet_pre_exit run #6: crashed: INFO: task hung in devlink_free run #7: crashed: INFO: task hung in del_device_store run #8: crashed: no output from test machine run #9: crashed: no output from test machine representative crash: INFO: task hung in devlink_pernet_pre_exit, types: [HANG] # git bisect bad d15121be7485655129101f3960ae6add40204463 d15121be7485655129101f3960ae6add40204463 is the first bad commit commit d15121be7485655129101f3960ae6add40204463 Author: Paolo Abeni Date: Mon May 8 08:17:44 2023 +0200 Revert "softirq: Let ksoftirqd do its job" This reverts the following commits: 4cd13c21b207 ("softirq: Let ksoftirqd do its job") 3c53776e29f8 ("Mark HI and TASKLET softirq synchronous") 1342d8080f61 ("softirq: Don't skip softirq execution when softirq thread is parking") in a single change to avoid known bad intermediate states introduced by a patch series reverting them individually. Due to the mentioned commit, when the ksoftirqd threads take charge of softirq processing, the system can experience high latencies. In the past a few workarounds have been implemented for specific side-effects of the initial ksoftirqd enforcement commit: commit 1ff688209e2e ("watchdog: core: make sure the watchdog_worker is not deferred") commit 8d5755b3f77b ("watchdog: softdog: fire watchdog even if softirqs do not get to run") commit 217f69743681 ("net: busy-poll: allow preemption in sk_busy_loop()") commit 3c53776e29f8 ("Mark HI and TASKLET softirq synchronous") But the latency problem still exists in real-life workloads, see the link below. The reverted commit intended to solve a live-lock scenario that can now be addressed with the NAPI threaded mode, introduced with commit 29863d41bb6e ("net: implement threaded-able napi poll loop support"), which is nowadays in a pretty stable status. While a complete solution to put softirq processing under nice resource control would be preferable, that has proven to be a very hard task. In the short term, remove the main pain point, and also simplify a bit the current softirq implementation. Signed-off-by: Paolo Abeni Signed-off-by: Thomas Gleixner Tested-by: Jason Xing Reviewed-by: Jakub Kicinski Reviewed-by: Eric Dumazet Reviewed-by: Sebastian Andrzej Siewior Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: netdev@vger.kernel.org Link: https://lore.kernel.org/netdev/305d7742212cbe98621b16be782b0562f1012cb6.camel@redhat.com Link: https://lore.kernel.org/r/57e66b364f1b6f09c9bc0316742c3b14f4ce83bd.1683526542.git.pabeni@redhat.com kernel/softirq.c | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-) accumulated error probability: 0.00 parent commit ac9a78681b921877518763ba0e89202254349d1b wasn't tested testing commit ac9a78681b921877518763ba0e89202254349d1b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 7244b1aedb6af184625504a9570a87fd82009afe75aabd3ff56cf7bff87d64a2 culprit signature: abc76c232677e537ee7dc088a1e92896b5208436e2fd5009184425a3168474e7 parent signature: 7244b1aedb6af184625504a9570a87fd82009afe75aabd3ff56cf7bff87d64a2 revisions tested: 28, total time: 10h34m19.276105828s (build: 6h13m5.024214926s, test: 4h1m52.618365506s) first bad commit: d15121be7485655129101f3960ae6add40204463 Revert "softirq: Let ksoftirqd do its job" recipients (to): ["bigeasy@linutronix.de" "edumazet@google.com" "kerneljasonxing@gmail.com" "kuba@kernel.org" "pabeni@redhat.com" "tglx@linutronix.de"] recipients (cc): [] crash: INFO: task hung in devlink_pernet_pre_exit INFO: task kworker/u4:5:3752 blocked for more than 143 seconds. Not tainted 6.4.0-rc1-syzkaller #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/u4:5 state:D stack:11592 pid:3752 ppid:2 flags:0x00004000 Workqueue: netns cleanup_net Call Trace: context_switch kernel/sched/core.c:5343 [inline] __schedule+0x6af/0x9c0 kernel/sched/core.c:6669 schedule+0x68/0xc0 kernel/sched/core.c:6745 schedule_preempt_disabled+0x32/0x60 kernel/sched/core.c:6804 __mutex_lock_common kernel/locking/mutex.c:679 [inline] __mutex_lock+0x449/0x970 kernel/locking/mutex.c:747 devl_lock net/devlink/core.c:54 [inline] devlink_pernet_pre_exit+0x63/0x1e0 net/devlink/core.c:288 ops_pre_exit_list net/core/net_namespace.c:160 [inline] cleanup_net+0x2b4/0x5e0 net/core/net_namespace.c:602 process_one_work+0x319/0x740 kernel/workqueue.c:2405 worker_thread+0x328/0x580 kernel/workqueue.c:2552 kthread+0x113/0x140 kernel/kthread.c:379 ret_from_fork+0x29/0x50 arch/x86/entry/entry_64.S:308