ci2 starts bisection 2025-05-20 18:11:55.89609461 +0000 UTC m=+26628.327905962 bisecting cause commit starting from a5806cd506af5a7c19bcd596e4708b5c464bfd21 building syzkaller on b47f9e02307162230709e7f183621732455244b0 ensuring issue is reproducible on original commit a5806cd506af5a7c19bcd596e4708b5c464bfd21 testing commit a5806cd506af5a7c19bcd596e4708b5c464bfd21 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: b5ee8c41782076d9232a425724910cf32bfecaff65c7a19fef94fe44112b42b2 all runs: crashed: kernel BUG in blk_mq_end_request representative crash: kernel BUG in blk_mq_end_request, types: [BUG] check whether we can drop unnecessary instrumentation disabling configs for [ATOMIC_SLEEP HANG LEAK UBSAN KASAN LOCKDEP], they are not needed testing commit a5806cd506af5a7c19bcd596e4708b5c464bfd21 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: cb12a4f6215e7cb2b392cd96287aba808ab8f4522e7f9732fb0e96aa64fecce0 all runs: crashed: kernel BUG in blk_mq_end_request representative crash: kernel BUG in blk_mq_end_request, types: [BUG] the bug reproduces without the instrumentation disabling configs for [HANG LEAK UBSAN KASAN LOCKDEP ATOMIC_SLEEP], they are not needed kconfig minimization: base=4084 full=8326 leaves diff=2130 split chunks (needed=false): <2130> split chunk #0 of len 2130 into 5 parts testing without sub-chunk 1/5 disabling configs for [HANG LEAK UBSAN KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit a5806cd506af5a7c19bcd596e4708b5c464bfd21 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: de78656822ee199a67c1e9779d9c44dc6c1f955949c9883f7210fb19a1010cfb all runs: OK false negative chance: 0.000 testing without sub-chunk 2/5 disabling configs for [LEAK UBSAN KASAN LOCKDEP ATOMIC_SLEEP HANG], they are not needed testing commit a5806cd506af5a7c19bcd596e4708b5c464bfd21 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 52b25aa3cae2ad22a123181aa2d301a2592260e5734ff75c5f6800e8e9d70796 all runs: crashed: kernel BUG in blk_mq_end_request representative crash: kernel BUG in blk_mq_end_request, types: [BUG] the chunk can be dropped testing without sub-chunk 3/5 disabling configs for [LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN KASAN], they are not needed testing commit a5806cd506af5a7c19bcd596e4708b5c464bfd21 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 833d605cf278e08982f2f81c282b828bac06246902a1d8331092cb844c96b1ca all runs: crashed: kernel BUG in blk_mq_end_request representative crash: kernel BUG in blk_mq_end_request, types: [BUG] the chunk can be dropped testing without sub-chunk 4/5 disabling configs for [LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN KASAN], they are not needed testing commit a5806cd506af5a7c19bcd596e4708b5c464bfd21 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 57422b8a8ea1c35561ec69f06d6d10fb671ef5151ce44e23bf93d5edba65c094 all runs: crashed: kernel BUG in blk_mq_end_request representative crash: kernel BUG in blk_mq_end_request, types: [BUG] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [ATOMIC_SLEEP HANG LEAK UBSAN KASAN LOCKDEP], they are not needed testing commit a5806cd506af5a7c19bcd596e4708b5c464bfd21 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 533e693c20dcec941f01706d94ade30cc38534bf63c3449d8c37ed891cc2f587 all runs: crashed: kernel BUG in blk_mq_end_request representative crash: kernel BUG in blk_mq_end_request, types: [BUG] the chunk can be dropped minimized to 426 configs; suspects: [6LOWPAN 6LOWPAN_GHC_EXT_HDR_DEST 6LOWPAN_GHC_EXT_HDR_FRAG 6LOWPAN_GHC_EXT_HDR_HOP 6LOWPAN_GHC_EXT_HDR_ROUTE 6LOWPAN_GHC_ICMPV6 6LOWPAN_GHC_UDP 6LOWPAN_NHC 6LOWPAN_NHC_DEST 6LOWPAN_NHC_FRAGMENT 6LOWPAN_NHC_HOP 6LOWPAN_NHC_IPV6 6LOWPAN_NHC_MOBILITY 6LOWPAN_NHC_ROUTING 6LOWPAN_NHC_UDP 6PACK 842_COMPRESS 842_DECOMPRESS 9P_FSCACHE 9P_FS_POSIX_ACL 9P_FS_SECURITY ACORN_PARTITION ACORN_PARTITION_ADFS ACORN_PARTITION_CUMANA ACORN_PARTITION_EESOX ACORN_PARTITION_ICS ACORN_PARTITION_POWERTEC ACORN_PARTITION_RISCIX ACPI_NFIT ACPI_NHLT ACPI_PLATFORM_PROFILE ADFS_FS AFFS_FS AFS_FS AFS_FSCACHE AF_KCM AF_RXRPC AF_RXRPC_IPV6 AIX_PARTITION AMD_SFH_HID AMIGA_PARTITION ANDROID_BINDERFS ANDROID_BINDER_IPC ANON_VMA_NAME APERTURE_HELPERS APPLE_MFI_FASTCHARGE AR5523 ARCH_ENABLE_MEMORY_HOTREMOVE ARCH_ENABLE_THP_MIGRATION ARCH_HAS_USER_SHADOW_STACK ARCH_SUPPORTS_HUGE_PFNMAP ARCH_SUPPORTS_PMD_PFNMAP ARCH_SUPPORTS_PUD_PFNMAP ARCH_WANT_PMD_MKWRITE ASM_MODVERSIONS ASUS_TF103C_DOCK ASYNC_CORE ASYNC_MEMCPY ASYNC_PQ ASYNC_RAID6_RECOV ASYNC_TX_DMA ASYNC_XOR AS_SHA1_NI AS_SHA256_NI AS_TPAUSE ATARI_PARTITION ATA_GENERIC ATA_OVER_ETH ATH10K ATH10K_CE ATH10K_LEDS ATH10K_PCI ATH10K_USB ATH11K ATH6KL ATH6KL_USB ATH9K ATH9K_AHB ATH9K_BTCOEX_SUPPORT ATH9K_CHANNEL_CONTEXT ATH9K_COMMON ATH9K_COMMON_DEBUG ATH9K_COMMON_SPECTRAL ATH9K_DEBUGFS ATH9K_DYNACK ATH9K_HTC ATH9K_HTC_DEBUGFS ATH9K_HW ATH9K_PCI ATH9K_PCOEM ATH9K_RFKILL ATH_COMMON ATM ATM_BR2684 ATM_CLIP ATM_DRIVERS ATM_LANE ATM_MPOA ATM_TCP AUXILIARY_BUS AX25 AX25_DAMA_SLAVE AX88796B_PHY BAREUDP BATMAN_ADV BATMAN_ADV_BATMAN_V BATMAN_ADV_BLA BATMAN_ADV_DAT BATMAN_ADV_MCAST BATMAN_ADV_NC BCACHE BCACHEFS_DEBUG BCACHEFS_ERASURE_CODING BCACHEFS_FS BCACHEFS_POSIX_ACL BCACHEFS_QUOTA BCACHEFS_SIX_OPTIMISTIC_SPIN BCMA BCMA_HOST_PCI_POSSIBLE BEFS_FS BFQ_CGROUP_DEBUG BFQ_GROUP_IOSCHED BFS_FS BIG_KEYS BLK_CGROUP_PUNT_BIO BLK_CGROUP_RWSTAT BLK_DEV_BSGLIB BLK_DEV_INTEGRITY BLK_DEV_NBD BLK_DEV_NULL_BLK BLK_DEV_NULL_BLK_FAULT_INJECTION BLK_DEV_NVME BLK_DEV_PMEM BLK_DEV_RAM BLK_DEV_RNBD BLK_DEV_RNBD_CLIENT BLK_DEV_THROTTLING BLK_DEV_ZONED BLK_ICQ BLK_INLINE_ENCRYPTION BLK_INLINE_ENCRYPTION_FALLBACK BLK_WBT BLK_WBT_MQ BONDING BOOT_VESA_SUPPORT BPF_EVENTS BPF_JIT BPF_JIT_ALWAYS_ON BPF_JIT_DEFAULT_ON BPF_LSM BPF_PRELOAD BPF_PRELOAD_UMD BPF_STREAM_PARSER BPF_SYSCALL BPQETHER BRIDGE BRIDGE_CFM BRIDGE_EBT_802_3 BRIDGE_EBT_AMONG BRIDGE_EBT_ARP BRIDGE_EBT_ARPREPLY BRIDGE_EBT_BROUTE BRIDGE_EBT_DNAT BRIDGE_EBT_IP BRIDGE_EBT_IP6 BRIDGE_EBT_LIMIT BRIDGE_EBT_LOG BRIDGE_EBT_MARK BRIDGE_EBT_MARK_T BRIDGE_EBT_NFLOG BRIDGE_EBT_PKTTYPE BRIDGE_EBT_REDIRECT BRIDGE_EBT_SNAT BRIDGE_EBT_STP BRIDGE_EBT_T_FILTER BRIDGE_EBT_T_NAT BRIDGE_EBT_VLAN BRIDGE_IGMP_SNOOPING BRIDGE_MRP BRIDGE_NF_EBTABLES BRIDGE_NF_EBTABLES_LEGACY BRIDGE_VLAN_FILTERING BSD_DISKLABEL BSD_PROCESS_ACCT_V3 BT BTRFS_ASSERT BTRFS_FS BTRFS_FS_POSIX_ACL BTRFS_FS_REF_VERIFY BTT BT_6LOWPAN BT_ATH3K BT_BCM BT_BNEP BT_BNEP_MC_FILTER BT_BNEP_PROTO_FILTER BT_BREDR BT_HCIBCM203X BT_HCIBFUSB BT_HCIBPA10X BT_HCIBTUSB BT_HCIBTUSB_AUTOSUSPEND BT_HCIBTUSB_BCM BT_HCIBTUSB_MTK BT_HCIBTUSB_POLL_SYNC BT_HCIBTUSB_RTL BT_HCIUART BT_HCIUART_3WIRE BT_HCIUART_AG6XX BT_HCIUART_BCSP BT_HCIUART_H4 BT_HCIUART_LL BT_HCIUART_MRVL BT_HCIUART_QCA BT_HCIUART_SERDEV BT_HCIVHCI BT_INTEL BT_LE BT_LEDS BT_LE_L2CAP_ECRED BT_MRVL BT_MRVL_SDIO BT_MSFTEXT BT_MTK BT_MTKSDIO BT_MTKUART BT_QCA BT_RFCOMM BT_RFCOMM_TTY BT_RTL CACHEFILES CAIF CAIF_DEBUG CAIF_DRIVERS CAIF_NETDEV CAIF_TTY CAIF_USB CAIF_VIRTIO CAN CAN_8DEV_USB CAN_BCM CAN_CALC_BITTIMING CAN_DEV CAN_EMS_USB CAN_ESD_USB CAN_ETAS_ES58X CAN_F81604 CAN_GS_USB CAN_GW CAN_IFI_CANFD CAN_ISOTP CAN_J1939 CAN_KVASER_USB CAN_MCBA_USB CAN_NETLINK CAN_PEAK_USB CAN_RAW CAN_RX_OFFLOAD CAN_SLCAN CAN_UCAN CAN_VCAN CAN_VXCAN CARL9170 CARL9170_HWRNG CARL9170_LEDS CARL9170_WPC CEC_CORE CEPH_FS CEPH_FSCACHE CEPH_FS_POSIX_ACL CEPH_LIB CEPH_LIB_USE_DNS_RESOLVER CFG80211 CFG80211_CRDA_SUPPORT CFG80211_DEBUGFS CFG80211_DEFAULT_PS CFG80211_REQUIRE_SIGNED_REGDB CFG80211_USE_KERNEL_REGDB_KEYS CFG80211_WEXT CFS_BANDWIDTH CGROUP_BPF CHARGER_ISP1704 CHR_DEV_ST CIFS CIFS_ALLOW_INSECURE_LEGACY CIFS_DEBUG CIFS_DFS_UPCALL CIFS_FSCACHE CIFS_POSIX CIFS_SMB_DIRECT CIFS_SWN_UPCALL CIFS_UPCALL CIFS_XATTR CLOSURES CLS_U32_MARK CLS_U32_PERF CMA CMA_SIZE_SEL_MAX CMDLINE_PARTITION COMEDI COMEDI_DT9812 COMEDI_NI_USB6501 COMEDI_USBDUX COMEDI_USBDUXFAST COMEDI_USBDUXSIGMA COMEDI_USB_DRIVERS COMEDI_VMK80XX COMPAT_NETLINK_MESSAGES COUNTER CRAMFS CRAMFS_BLOCKDEV CRAMFS_MTD CRC64 CRC64_ARCH CRC8 CRC_ITU_T CRC_T10DIF CRC_T10DIF_ARCH CRYPTO_842 CRYPTO_ADIANTUM CRYPTO_AEGIS128 CRYPTO_AEGIS128_AESNI_SSE2 CRYPTO_AES_NI_INTEL CRYPTO_AES_TI CRYPTO_ANSI_CPRNG CRYPTO_ANUBIS CRYPTO_ARC4 CRYPTO_ARCH_HAVE_LIB_BLAKE2S CRYPTO_ARCH_HAVE_LIB_CHACHA CRYPTO_ARCH_HAVE_LIB_CURVE25519 CRYPTO_ARCH_HAVE_LIB_POLY1305 CRYPTO_ARIA CRYPTO_ARIA_AESNI_AVX_X86_64 CRYPTO_BLAKE2B CRYPTO_BLAKE2S_X86 CRYPTO_BLOWFISH CRYPTO_BLOWFISH_COMMON CRYPTO_BLOWFISH_X86_64 CRYPTO_CAMELLIA CRYPTO_CAMELLIA_AESNI_AVX2_X86_64 CRYPTO_CAMELLIA_AESNI_AVX_X86_64 CRYPTO_CAMELLIA_X86_64 CRYPTO_CAST5 CRYPTO_CAST5_AVX_X86_64 CRYPTO_CAST6 CRYPTO_CAST6_AVX_X86_64 CRYPTO_CAST_COMMON CRYPTO_CHACHA20 CRYPTO_CHACHA20POLY1305 CRYPTO_CHACHA20_X86_64 CRYPTO_CRC32C CRYPTO_CRYPTD CRYPTO_CTS CRYPTO_CURVE25519 CRYPTO_CURVE25519_X86 CRYPTO_DEFLATE CRYPTO_DES CRYPTO_DES3_EDE_X86_64 CRYPTO_DEV_CCP CRYPTO_DEV_CCP_DD CRYPTO_DEV_PADLOCK CRYPTO_DEV_PADLOCK_AES CRYPTO_DEV_PADLOCK_SHA CRYPTO_DEV_QAT CRYPTO_DEV_QAT_C3XXX CRYPTO_DEV_QAT_C3XXXVF CRYPTO_DEV_QAT_C62X CRYPTO_DEV_QAT_C62XVF CRYPTO_DEV_QAT_DH895xCC CRYPTO_DEV_QAT_DH895xCCVF CRYPTO_DEV_VIRTIO CRYPTO_DH CRYPTO_DRBG_CTR CRYPTO_DRBG_HASH CRYPTO_ECDH CRYPTO_ECRDSA CRYPTO_ENGINE CRYPTO_ESSIV CRYPTO_FCRYPT CRYPTO_GHASH_CLMUL_NI_INTEL CRYPTO_HCTR2 CRYPTO_HKDF CRYPTO_KDF800108_CTR CRYPTO_KHAZAD CRYPTO_KPP CRYPTO_KRB5 CRYPTO_KRB5ENC CRYPTO_LIB_ARC4 CRYPTO_LIB_CHACHA CRYPTO_LIB_CHACHA20POLY1305 CRYPTO_LIB_CHACHA_GENERIC CRYPTO_LIB_CHACHA_INTERNAL CRYPTO_LIB_CURVE25519 CRYPTO_LIB_CURVE25519_GENERIC CRYPTO_LIB_CURVE25519_INTERNAL CRYPTO_LIB_DES CRYPTO_LIB_POLY1305 CRYPTO_LIB_POLY1305_GENERIC CRYPTO_LIB_POLY1305_INTERNAL CRYPTO_LRW CRYPTO_LZ4 CRYPTO_LZ4HC CRYPTO_MICHAEL_MIC CRYPTO_NHPOLY1305 CRYPTO_NHPOLY1305_AVX2 CRYPTO_NHPOLY1305_SSE2 CRYPTO_PCBC CRYPTO_PCRYPT CRYPTO_POLY1305 CRYPTO_POLY1305_X86_64 CRYPTO_POLYVAL CRYPTO_POLYVAL_CLMUL_NI CRYPTO_RMD160 CRYPTO_SEED CRYPTO_SERPENT CRYPTO_SERPENT_AVX2_X86_64 CRYPTO_SERPENT_AVX_X86_64 CRYPTO_SERPENT_SSE2_X86_64 CRYPTO_SHA1_SSSE3 CRYPTO_SHA256_SSSE3 CRYPTO_SHA512_SSSE3 CRYPTO_SIMD CRYPTO_SM3 CRYPTO_SM3_AVX_X86_64 CRYPTO_SM4 CRYPTO_SM4_AESNI_AVX2_X86_64 CRYPTO_SM4_AESNI_AVX_X86_64 CRYPTO_SM4_GENERIC CRYPTO_STREEBOG CRYPTO_TEA CRYPTO_TWOFISH CRYPTO_TWOFISH_AVX_X86_64 CRYPTO_TWOFISH_COMMON CRYPTO_TWOFISH_X86_64 CRYPTO_TWOFISH_X86_64_3WAY CRYPTO_USER CRYPTO_USER_API CRYPTO_USER_API_AEAD CRYPTO_USER_API_ENABLE_OBSOLETE CRYPTO_USER_API_HASH CRYPTO_USER_API_RNG CRYPTO_USER_API_SKCIPHER CRYPTO_WP512 CRYPTO_XCBC CRYPTO_XCTR CRYPTO_XTS CRYPTO_XXHASH CRYPTO_ZSTD CUSE CYPRESS_FIRMWARE DAMON DAMON_PADDR DAMON_RECLAIM DAMON_VADDR DAX DCA DCB DEBUG_VFS DEFAULT_CODEL DEVICE_MIGRATION DEVICE_PRIVATE DEV_COREDUMP DEV_DAX DLN2_ADC DMABUF_HEAPS DMABUF_HEAPS_CMA DMABUF_HEAPS_SYSTEM DMABUF_MOVE_NOTIFY DMA_CMA DMA_ENGINE_RAID DM_AUDIT DM_BIO_PRISON DM_BUFIO DM_CACHE DM_CACHE_SMQ DM_CLONE DM_CRYPT DM_FLAKEY DM_INTEGRITY DM_MULTIPATH DM_MULTIPATH_QL DM_MULTIPATH_ST DM_PERSISTENT_DATA DM_RAID DM_SNAPSHOT DM_THIN_PROVISIONING DM_UEVENT DM_VERITY DM_VERITY_FEC DM_WRITECACHE DM_ZONED DRAGONRISE_FF DRM DRM_AUX_BRIDGE DRM_BOCHS DRM_BRIDGE DRM_BUDDY DRM_CIRRUS_QEMU DRM_CLIENT DRM_CLIENT_DEFAULT_FBDEV DRM_CLIENT_LIB DRM_CLIENT_SELECTION DRM_CLIENT_SETUP DRM_FBDEV_EMULATION ENCRYPTED_KEYS FSCACHE FUSE_FS GPIOLIB HAMRADIO HID_DRAGONRISE IIO INFINIBAND INFINIBAND_ADDR_TRANS INFINIBAND_RTRS_CLIENT IOSCHED_BFQ LIBNVDIMM MAC80211 MAC80211_DEBUGFS MAC80211_LEDS MEDIA_SUPPORT MEMORY_HOTPLUG MEMORY_HOTREMOVE MFD_DLN2 MMC MTD NET_CLS_U32 NET_SCH_DEFAULT PARTITION_ADVANCED RFKILL SERIAL_DEV_BUS TLS TLS_DEVICE TRANSPARENT_HUGEPAGE TRUSTED_KEYS USB_GADGET USB_PHY VLAN_8021Q WANT_COMPAT_NETLINK_MESSAGES WEXT_CORE WIRELESS WLAN WLAN_VENDOR_ATH ZONE_DEVICE] disabling configs for [ATOMIC_SLEEP HANG LEAK UBSAN KASAN LOCKDEP], they are not needed picked [v6.14 v6.13 v6.12 v6.10 v6.8 v6.6 v6.4 v6.2 v5.19 v5.16 v5.13 v5.10 v5.7 v5.4 v5.1 v4.19] out of 37 release tags testing release v6.14 testing commit 38fec10eb60d687e30c8c6b5420d86e8149f7557 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 980d77db1d58f4711e4745c6cd69eaba4681f17eafbc57a5965fc619fbed9cc1 all runs: crashed: kernel BUG in blk_mq_end_request representative crash: kernel BUG in blk_mq_end_request, types: [BUG] testing release v6.13 testing commit ffd294d346d185b70e28b1a28abe367bbfe53c04 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: d746dfe18bbbee57beec90e9a912adf354815b0163d021fdb54f0c9d8a78411b all runs: crashed: kernel BUG in blk_mq_end_request representative crash: kernel BUG in blk_mq_end_request, types: [BUG] testing release v6.12 testing commit adc218676eef25575469234709c2d87185ca223a gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 315acd7c4c71e1369d20c697bdd52c0afdcd42fa285f2643d1d1b836681fa348 all runs: crashed: kernel BUG in blk_mq_end_request representative crash: kernel BUG in blk_mq_end_request, types: [BUG] testing release v6.10 testing commit 0c3836482481200ead7b416ca80c68a29cfdaabd gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 54d8535d8bb64343033eaeb61f23bd0749c0c5b7628d64f478e6895cadefefd7 all runs: OK false negative chance: 0.000 # git bisect start adc218676eef25575469234709c2d87185ca223a 0c3836482481200ead7b416ca80c68a29cfdaabd Bisecting: 14868 revisions left to test after this (roughly 14 steps) [703896be3015db7f8fd8822b18909a5914209a70] Merge tag 'sound-6.11-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound testing commit 703896be3015db7f8fd8822b18909a5914209a70 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 6b436b92c924eb66899e0137f8dda39528e038a387a1284dae93d6de0e2da51d all runs: crashed: BUG: workqueue lockup representative crash: BUG: workqueue lockup, types: [UNKNOWN] # git bisect bad 703896be3015db7f8fd8822b18909a5914209a70 Bisecting: 6814 revisions left to test after this (roughly 13 steps) [b3ce7a30847a54a7f96a35e609303d8afecd460b] Merge tag 'drm-next-2024-07-18' of https://gitlab.freedesktop.org/drm/kernel testing commit b3ce7a30847a54a7f96a35e609303d8afecd460b gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 2676cb7c263f2da4185315f616af81ac34464da016a5faef55b2468c80eda3b2 all runs: OK false negative chance: 0.000 # git bisect good b3ce7a30847a54a7f96a35e609303d8afecd460b Bisecting: 3536 revisions left to test after this (roughly 12 steps) [7846b618e0a4c3e08888099d1d4512722b39ca99] Merge tag 'rtc-6.11' of git://git.kernel.org/pub/scm/linux/kernel/git/abelloni/linux testing commit 7846b618e0a4c3e08888099d1d4512722b39ca99 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 3b80590994b138630b16ba17f3a26d9711a4865833aa70e3308c9f3d3052cc7e all runs: crashed: BUG: workqueue lockup representative crash: BUG: workqueue lockup, types: [UNKNOWN] # git bisect bad 7846b618e0a4c3e08888099d1d4512722b39ca99 Bisecting: 1639 revisions left to test after this (roughly 11 steps) [53a5182c8a6805d3096336709ba5790d16f8c369] Merge tag 'for-6.11-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux testing commit 53a5182c8a6805d3096336709ba5790d16f8c369 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 8d22eeec26a1903f8f1def47f3fba49177a57d13ddabfae93cf1b6380690df9d all runs: crashed: BUG: workqueue lockup representative crash: BUG: workqueue lockup, types: [UNKNOWN] # git bisect bad 53a5182c8a6805d3096336709ba5790d16f8c369 Bisecting: 825 revisions left to test after this (roughly 10 steps) [3d51520954154a476bfdacf9427acd1d9538734c] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma testing commit 3d51520954154a476bfdacf9427acd1d9538734c gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 960e6e2c8afcd65165d945f5a094507eb04f096cdf29a4ea8e174f7302b2b8a7 all runs: crashed: BUG: workqueue lockup representative crash: BUG: workqueue lockup, types: [UNKNOWN] # git bisect bad 3d51520954154a476bfdacf9427acd1d9538734c Bisecting: 406 revisions left to test after this (roughly 9 steps) [dddebdece62ead1ac1112e6df375f56a1cb45f84] Merge tag 'vfs-6.11-rc1.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs testing commit dddebdece62ead1ac1112e6df375f56a1cb45f84 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 717a1d333ecb21c0c052f6897ce9760e348ade8bdbfd43f49bf08d55cd0071d3 all runs: OK false negative chance: 0.000 # git bisect good dddebdece62ead1ac1112e6df375f56a1cb45f84 Bisecting: 180 revisions left to test after this (roughly 8 steps) [720261cfc7329406a50c2a8536e0039b9dd9a4e5] Merge tag 'bcachefs-2024-07-18.2' of https://evilpiepirate.org/git/bcachefs testing commit 720261cfc7329406a50c2a8536e0039b9dd9a4e5 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 255d7e531e19f917ad6fa399e88fd3be3ea2858e62767d3f75c0bef7f7227844 all runs: crashed: BUG: workqueue lockup representative crash: BUG: workqueue lockup, types: [UNKNOWN] # git bisect bad 720261cfc7329406a50c2a8536e0039b9dd9a4e5 Bisecting: 112 revisions left to test after this (roughly 7 steps) [efb2018e4d238cc205690ac62c0917d60d291e66] bcachefs: Kill bch2_assert_btree_nodes_not_locked() testing commit efb2018e4d238cc205690ac62c0917d60d291e66 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 6afbaffc3958ea18eeb6cd38d5c6b5e13d13b77cd4990b905eee99f3c24aa262 all runs: crashed: BUG: workqueue lockup representative crash: BUG: workqueue lockup, types: [UNKNOWN] # git bisect bad efb2018e4d238cc205690ac62c0917d60d291e66 Bisecting: 56 revisions left to test after this (roughly 6 steps) [9ab55df5991eb967f6b94f8e1b855ffacdc50668] bcachefs: Walk leaf to root in btree_gc testing commit 9ab55df5991eb967f6b94f8e1b855ffacdc50668 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 2c0601eb05752646199988d3acb2fc3677101798606039903de445e39333aa41 all runs: crashed: BUG: workqueue lockup representative crash: BUG: workqueue lockup, types: [UNKNOWN] # git bisect bad 9ab55df5991eb967f6b94f8e1b855ffacdc50668 Bisecting: 27 revisions left to test after this (roughly 5 steps) [2744e5c9eb1a1090b5f61c955e934c70bfe6b04c] bcachefs: KEY_TYPE_accounting testing commit 2744e5c9eb1a1090b5f61c955e934c70bfe6b04c gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: f6b9b7e72971ed5ba2b218186ab12927d52722a1f30086f1c9865242adefdc79 all runs: OK false negative chance: 0.000 # git bisect good 2744e5c9eb1a1090b5f61c955e934c70bfe6b04c Bisecting: 13 revisions left to test after this (roughly 4 steps) [fb23d57a6dfc4e521c003dc542799f07d22d269e] bcachefs: Convert gc to new accounting testing commit fb23d57a6dfc4e521c003dc542799f07d22d269e gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 8295b48804c4377575e36b8e3bc18c0339d27fa8eb57f1ebdf721a2985462829 all runs: crashed: BUG: workqueue lockup representative crash: BUG: workqueue lockup, types: [UNKNOWN] # git bisect bad fb23d57a6dfc4e521c003dc542799f07d22d269e Bisecting: 6 revisions left to test after this (roughly 3 steps) [6b39638b84b462e29e8008f5b5f6fad93e021544] bcachefs: Convert bch2_ioctl_fs_usage() to new accounting testing commit 6b39638b84b462e29e8008f5b5f6fad93e021544 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: 411c2d2149fb51ce5093ea8665030c7ccc1314e172158e9e0a950be83823036c all runs: crashed: BUG: workqueue lockup representative crash: BUG: workqueue lockup, types: [UNKNOWN] # git bisect bad 6b39638b84b462e29e8008f5b5f6fad93e021544 Bisecting: 3 revisions left to test after this (roughly 2 steps) [1d16c605cc55ef26f0c65b362665a6c99080ccbc] bcachefs: Disk space accounting rewrite testing commit 1d16c605cc55ef26f0c65b362665a6c99080ccbc gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: e76b18e08044a16454fdec393d048b07e9bd7dbe8a5147239d802c9b4bc314b5 all runs: OK false negative chance: 0.000 # git bisect good 1d16c605cc55ef26f0c65b362665a6c99080ccbc Bisecting: 1 revision left to test after this (roughly 1 step) [f5095b9f85a1674a92d00e7ab466499a8ba49ce1] bcachefs: dev_usage updated by new accounting testing commit f5095b9f85a1674a92d00e7ab466499a8ba49ce1 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: c50439827a958a5e1a7a66204fa99e4e80f130218042b4546599c9e4ca5721ef all runs: crashed: BUG: workqueue lockup representative crash: BUG: workqueue lockup, types: [UNKNOWN] # git bisect bad f5095b9f85a1674a92d00e7ab466499a8ba49ce1 Bisecting: 0 revisions left to test after this (roughly 0 steps) [2e8d686a4a13c01d9a2b329507a0f5ce6455b5a8] bcachefs: Coalesce accounting keys before journal replay testing commit 2e8d686a4a13c01d9a2b329507a0f5ce6455b5a8 gcc compiler: Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6 kernel signature: f814b71312865b423e4636a76bae557579f884e2d51f25fc8d3fbc8062614b54 all runs: OK false negative chance: 0.000 # git bisect good 2e8d686a4a13c01d9a2b329507a0f5ce6455b5a8 f5095b9f85a1674a92d00e7ab466499a8ba49ce1 is the first bad commit commit f5095b9f85a1674a92d00e7ab466499a8ba49ce1 Author: Kent Overstreet Date: Mon Jan 1 19:42:37 2024 -0500 bcachefs: dev_usage updated by new accounting Reading disk accounting now requires an eytzinger lookup (see: bch2_accounting_mem_read()), but the per-device counters are used frequently enough that we'd like to still be able to read them with just a percpu sum, as in the old code. This patch special cases the device counters; when we update in-memory accounting we also update the old style percpu counters if it's a deice counter update. Signed-off-by: Kent Overstreet fs/bcachefs/bcachefs.h | 3 +-- fs/bcachefs/btree_gc.c | 2 +- fs/bcachefs/buckets.c | 36 +++++------------------------------- fs/bcachefs/buckets_types.h | 2 +- fs/bcachefs/disk_accounting.c | 15 +++++++++++++++ fs/bcachefs/disk_accounting.h | 14 +++++++++++++- fs/bcachefs/recovery.c | 17 ----------------- fs/bcachefs/sb-clean.c | 17 ----------------- 8 files changed, 36 insertions(+), 70 deletions(-) accumulated error probability: 0.00 culprit signature: c50439827a958a5e1a7a66204fa99e4e80f130218042b4546599c9e4ca5721ef parent signature: f814b71312865b423e4636a76bae557579f884e2d51f25fc8d3fbc8062614b54 revisions tested: 26, total time: 8h35m31.124674955s (build: 4h24m11.840410731s, test: 3h45m14.68864723s) first bad commit: f5095b9f85a1674a92d00e7ab466499a8ba49ce1 bcachefs: dev_usage updated by new accounting recipients (to): ["kent.overstreet@linux.dev"] recipients (cc): [] crash: BUG: workqueue lockup BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 221s! BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=-20 stuck for 215s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: vmstat_shepherd, kfree_rcu_monitor, free_obj_work workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x4 nice=0 active=4 refcnt=5 in-flight: 11:fsnotify_mark_destroy_workfn fsnotify_mark_destroy_workfn ,823:fsnotify_connector_destroy_workfn fsnotify_connector_destroy_workfn workqueue events_power_efficient: flags=0x80 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=5 refcnt=6 pending: do_cache_clean, neigh_periodic_work, 2*neigh_managed_work, neigh_periodic_work workqueue rcu_gp: flags=0x8 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=10 refcnt=11 pending: 10*srcu_invoke_callbacks workqueue mm_percpu_wq: flags=0x8 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=4 pending: vmstat_update, lru_add_drain_per_cpu BAR(33) workqueue kblockd: flags=0x18 pwq 3: cpus=0 node=0 flags=0x0 nice=-20 active=2 refcnt=3 pending: 2*blk_mq_timeout_work workqueue dm_bufio_cache: flags=0x8 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: work_fn pool 8: cpus=0-1 flags=0x4 nice=0 hung=0s workers=9 idle: 717 133 352 40 13 1279 1110 Showing backtraces of running workers in stalled CPU-bound worker pools: