ci starts bisection 2023-09-26 12:53:20.253116465 +0000 UTC m=+1129744.272240473 bisecting cause commit starting from a59addacf899b1b21a7b7449a1c52c98704c2472 building syzkaller on 0b6a67ac4b0dc26f43030c5edd01c9175f13b784 ensuring issue is reproducible on original commit a59addacf899b1b21a7b7449a1c52c98704c2472 testing commit a59addacf899b1b21a7b7449a1c52c98704c2472 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9400db12174f81db563ddd76a0bf63872125f8cbfbf385da5fb629e131f1e67b all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] check whether we can drop unnecessary instrumentation disabling configs for [UBSAN KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK], they are not needed testing commit a59addacf899b1b21a7b7449a1c52c98704c2472 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 71dda4cb2e9da736d9a692b298aaba4797cc88f0f593b081b3ac7045e62983ff all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] the bug reproduces without the instrumentation disabling configs for [UBSAN KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK], they are not needed kconfig minimization: base=3883 full=7693 leaves diff=2018 split chunks (needed=false): <2018> split chunk #0 of len 2018 into 5 parts testing without sub-chunk 1/5 disabling configs for [UBSAN KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK], they are not needed testing commit a59addacf899b1b21a7b7449a1c52c98704c2472 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2852d556cb2a9d4a695bcb7ae28240e2cb4528e6d13ad35896f02ce02220db7c all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [HANG LEAK UBSAN KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit a59addacf899b1b21a7b7449a1c52c98704c2472 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d1f28adfe373b4a0b1b92060cbd10576cbd8cefea86257fe4e5452fae29185b2 all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] the chunk can be dropped testing without sub-chunk 3/5 disabling configs for [HANG LEAK UBSAN KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit a59addacf899b1b21a7b7449a1c52c98704c2472 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: eb1362f053f65fad064d538e160945925694df072f2eb1d34bd20921e2eabbe5 all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] the chunk can be dropped testing without sub-chunk 4/5 disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing commit a59addacf899b1b21a7b7449a1c52c98704c2472 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9d5df669e92e57ac30398ea98a2db265e2426bce6c78d1a15e372cc5e39ca872 all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [LEAK UBSAN KASAN LOCKDEP ATOMIC_SLEEP HANG], they are not needed testing commit a59addacf899b1b21a7b7449a1c52c98704c2472 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 44ed024ff741e34dcec34e059a8aeccc2f408cdb66c578e13940a5d5239e538d all runs: OK false negative chance: 0.000 minimized to 402 configs; suspects: [ARCH_ENABLE_MEMORY_HOTREMOVE ATM BCMA BLK_DEV_ZONED BPF_SYSCALL CARDBUS CFG80211 CFG80211_WEXT CMA COMMON_CLK CONTIG_ALLOC CRYPTO_842 CRYPTO_LZ4 CRYPTO_LZ4HC CRYPTO_LZO CRYPTO_ZSTD DVB_CORE EXTCON FB GPIOLIB HID_ZEROPLUS I2C_MUX IIO IOMMUFD IRQ_REMAP KVM KVM_INTEL LIBNVDIMM MEDIA_ANALOG_TV_SUPPORT MEDIA_CAMERA_SUPPORT MEDIA_CEC_SUPPORT MEDIA_CONTROLLER MEDIA_DIGITAL_TV_SUPPORT MEDIA_RADIO_SUPPORT MEDIA_SDR_SUPPORT MEDIA_SUPPORT MEDIA_TEST_SUPPORT MEDIA_USB_SUPPORT MEMORY_HOTPLUG MEMORY_HOTREMOVE MFD_VIPERBOARD PARPORT PCCARD PCMCIA PHONET RADIO_ADAPTERS RADIO_SI470X RADIO_SI4713 RC_CORE RFKILL SND SOUND SPI SSB TAP TARGET_CORE TUN USB_AMD5536UDC USB_ATM USB_CONFIGFS USB_CONFIGFS_F_MIDI USB_CONFIGFS_F_PRINTER USB_CONFIGFS_F_TCM USB_CONFIGFS_F_UAC1 USB_CONFIGFS_F_UAC1_LEGACY USB_CONFIGFS_F_UAC2 USB_CONFIGFS_F_UVC USB_CONFIGFS_MASS_STORAGE USB_CONFIGFS_NCM USB_CONFIGFS_OBEX USB_CONFIGFS_PHONET USB_CONFIGFS_RNDIS USB_CONFIGFS_SERIAL USB_CXACRU USB_CYPRESS_CY7C63 USB_CYTHERM USB_DSBR USB_DUMMY_HCD USB_DWC2 USB_DWC2_HOST USB_DWC2_PCI USB_DWC3 USB_DWC3_GADGET USB_DWC3_OF_SIMPLE USB_DWC3_PCI USB_DWC3_ULPI USB_DYNAMIC_MINORS USB_EG20T USB_EHCI_HCD_PLATFORM USB_EHCI_ROOT_HUB_TT USB_EHSET_TEST_FIXTURE USB_EMI26 USB_EMI62 USB_EPSON2888 USB_EZUSB_FX2 USB_FEW_INIT_RETRIES USB_F_ACM USB_F_ECM USB_F_EEM USB_F_FS USB_F_HID USB_F_MASS_STORAGE USB_F_MIDI USB_F_NCM USB_F_OBEX USB_F_PHONET USB_F_PRINTER USB_F_RNDIS USB_F_SERIAL USB_F_SS_LB USB_F_SUBSET USB_F_TCM USB_F_UAC1 USB_F_UAC1_LEGACY USB_F_UAC2 USB_F_UVC USB_GADGET USB_GADGETFS USB_GADGET_DEBUG_FILES USB_GADGET_DEBUG_FS USB_GL860 USB_GOKU USB_GPIO_VBUS USB_GR_UDC USB_GSPCA USB_GSPCA_BENQ USB_GSPCA_CONEX USB_GSPCA_CPIA1 USB_GSPCA_DTCS033 USB_GSPCA_ETOMS USB_GSPCA_FINEPIX USB_GSPCA_JEILINJ USB_GSPCA_JL2005BCD USB_GSPCA_KINECT USB_GSPCA_KONICA USB_GSPCA_MARS USB_GSPCA_MR97310A USB_GSPCA_NW80X USB_GSPCA_OV519 USB_GSPCA_OV534 USB_GSPCA_OV534_9 USB_GSPCA_PAC207 USB_GSPCA_PAC7302 USB_GSPCA_PAC7311 USB_GSPCA_SE401 USB_GSPCA_SN9C2028 USB_GSPCA_SN9C20X USB_GSPCA_SONIXB USB_GSPCA_SONIXJ USB_GSPCA_SPCA1528 USB_GSPCA_SPCA500 USB_GSPCA_SPCA501 USB_GSPCA_SPCA505 USB_GSPCA_SPCA506 USB_GSPCA_SPCA508 USB_GSPCA_SPCA561 USB_GSPCA_SQ905 USB_GSPCA_SQ905C USB_GSPCA_SQ930X USB_GSPCA_STK014 USB_GSPCA_STK1135 USB_GSPCA_STV0680 USB_GSPCA_SUNPLUS USB_GSPCA_T613 USB_GSPCA_TOPRO USB_GSPCA_TOUPTEK USB_GSPCA_TV8532 USB_GSPCA_VC032X USB_GSPCA_VICAM USB_GSPCA_XIRLINK_CIT USB_GSPCA_ZC3XX USB_HACKRF USB_HCD_BCMA USB_HCD_SSB USB_HSIC_USB3503 USB_HSIC_USB4604 USB_HSO USB_HUB_USB251XB USB_IDMOUSE USB_IOWARRIOR USB_IPHETH USB_ISIGHTFW USB_ISP116X_HCD USB_ISP1301 USB_ISP1760 USB_ISP1760_DUAL_ROLE USB_ISP1760_HCD USB_ISP1761_UDC USB_KAWETH USB_KC2190 USB_KEENE USB_LAN78XX USB_LCD USB_LD USB_LEDS_TRIGGER_USBPORT USB_LED_TRIG USB_LEGOTOWER USB_LIBCOMPOSITE USB_LINK_LAYER_TEST USB_M5602 USB_MA901 USB_MAX3421_HCD USB_MDC800 USB_MICROTEK USB_MR800 USB_MSI2500 USB_MUSB_DUAL_ROLE USB_MUSB_HDRC USB_MV_U3D USB_MV_UDC USB_NET2272 USB_NET2272_DMA USB_NET2280 USB_NET_AX88179_178A USB_NET_AX8817X USB_NET_CDCETHER USB_NET_CDC_EEM USB_NET_CDC_MBIM USB_NET_CDC_NCM USB_NET_CDC_SUBSET USB_NET_CDC_SUBSET_ENABLE USB_NET_CH9200 USB_NET_CX82310_ETH USB_NET_DM9601 USB_NET_GL620A USB_NET_HUAWEI_CDC_NCM USB_NET_INT51X1 USB_NET_KALMIA USB_NET_MCS7830 USB_NET_NET1080 USB_NET_PLUSB USB_NET_QMI_WWAN USB_NET_RNDIS_HOST USB_NET_RNDIS_WLAN USB_NET_SMSC75XX USB_NET_SMSC95XX USB_NET_SR9700 USB_NET_SR9800 USB_NET_ZAURUS USB_OHCI_HCD_PLATFORM USB_OTG USB_OTG_FSM USB_OXU210HP_HCD USB_PEGASUS USB_PULSE8_CEC USB_PWC USB_PWC_INPUT_EVDEV USB_PXA27X USB_R8A66597 USB_R8A66597_HCD USB_RAINSHADOW_CEC USB_RAREMONO USB_RAW_GADGET USB_RTL8150 USB_RTL8152 USB_RTL8153_ECM USB_S2255 USB_SERIAL USB_SERIAL_AIRCABLE USB_SERIAL_ARK3116 USB_SERIAL_BELKIN USB_SERIAL_CH341 USB_SERIAL_CONSOLE USB_SERIAL_CP210X USB_SERIAL_CYBERJACK USB_SERIAL_CYPRESS_M8 USB_SERIAL_DEBUG USB_SERIAL_DIGI_ACCELEPORT USB_SERIAL_EDGEPORT USB_SERIAL_EDGEPORT_TI USB_SERIAL_EMPEG USB_SERIAL_F81232 USB_SERIAL_F8153X USB_SERIAL_FTDI_SIO USB_SERIAL_GARMIN USB_SERIAL_GENERIC USB_SERIAL_IPAQ USB_SERIAL_IPW USB_SERIAL_IR USB_SERIAL_IUU USB_SERIAL_KEYSPAN USB_SERIAL_KEYSPAN_PDA USB_SERIAL_KLSI USB_SERIAL_KOBIL_SCT USB_SERIAL_MCT_U232 USB_SERIAL_METRO USB_SERIAL_MOS7715_PARPORT USB_SERIAL_MOS7720 USB_SERIAL_MOS7840 USB_SERIAL_MXUPORT USB_SERIAL_NAVMAN USB_SERIAL_OMNINET USB_SERIAL_OPTICON USB_SERIAL_OPTION USB_SERIAL_OTI6858 USB_SERIAL_PL2303 USB_SERIAL_QCAUX USB_SERIAL_QT2 USB_SERIAL_QUALCOMM USB_SERIAL_SAFE USB_SERIAL_SIERRAWIRELESS USB_SERIAL_SIMPLE USB_SERIAL_SPCP8X5 USB_SERIAL_SSU100 USB_SERIAL_SYMBOL USB_SERIAL_TI USB_SERIAL_UPD78F0730 USB_SERIAL_VISOR USB_SERIAL_WHITEHEAT USB_SERIAL_WISHBONE USB_SERIAL_WWAN USB_SERIAL_XR USB_SERIAL_XSENS_MT USB_SEVSEG USB_SI470X USB_SI4713 USB_SIERRA_NET USB_SISUSBVGA USB_SL811_CS USB_SL811_HCD USB_SL811_HCD_ISO USB_SNP_CORE USB_SPEEDTOUCH USB_STORAGE_ALAUDA USB_STORAGE_CYPRESS_ATACB USB_STORAGE_DATAFAB USB_STORAGE_ENE_UB6250 USB_STORAGE_FREECOM USB_STORAGE_ISD200 USB_STORAGE_JUMPSHOT USB_STORAGE_KARMA USB_STORAGE_ONETOUCH USB_STORAGE_SDDR09 USB_STORAGE_SDDR55 USB_STORAGE_USBAT USB_STV06XX USB_TEST USB_TMC USB_TRANCEVIBRATOR USB_UAS USB_UEAGLEATM USB_ULPI_BUS USB_USBNET USB_USS720 USB_U_AUDIO USB_U_ETHER USB_U_SERIAL USB_VIDEO_CLASS USB_VIDEO_CLASS_INPUT_EVDEV USB_VL600 USB_WDM USB_XHCI_DBGCAP USB_XHCI_PLATFORM USB_XUSBATM USB_YUREX USERFAULTFD USERIO USERMODE_DRIVER USER_RETURN_NOTIFIER UVC_COMMON U_SERIAL_CONSOLE V4L2_ASYNC V4L2_FWNODE V4L2_MEM2MEM_DEV V4L_TEST_DRIVERS VALIDATE_FS_PARSER VDPA VDPA_SIM VDPA_SIM_BLOCK VDPA_SIM_NET VDPA_USER VETH VFIO VFIO_GROUP VFIO_PCI VFIO_PCI_CORE VFIO_PCI_INTX VFIO_PCI_MMAP VFIO_VIRQFD VGASTATE VHOST VHOST_CROSS_ENDIAN_LEGACY VHOST_IOTLB VHOST_NET VHOST_RING VHOST_TASK VHOST_VDPA VHOST_VSOCK VIDEOBUF2_CORE VIDEOBUF2_DMA_CONTIG VIDEOBUF2_DMA_SG VIDEOBUF2_MEMOPS VIDEOBUF2_V4L2 VIDEOBUF2_VMALLOC VIDEOMODE_HELPERS VIDEO_AU0828 VIDEO_AU0828_RC VIDEO_AU0828_V4L2 VIDEO_CAMERA_SENSOR VIDEO_CMDLINE VIDEO_CS53L32A VIDEO_CX231XX VIDEO_CX231XX_ALSA VIDEO_CX231XX_DVB VIDEO_CX231XX_RC VIDEO_CX2341X VIDEO_CX25840 VIDEO_DEV VIDEO_EM28XX VIDEO_EM28XX_ALSA VIDEO_EM28XX_DVB VIDEO_EM28XX_RC VIDEO_EM28XX_V4L2 VIDEO_GO7007 VIDEO_GO7007_LOADER VIDEO_GO7007_USB VIDEO_GO7007_USB_S2250_BOARD VIDEO_HDPVR VIDEO_MSP3400 VIDEO_NOMODESET VIDEO_PVRUSB2 VIDEO_PVRUSB2_DVB VIDEO_PVRUSB2_SYSFS VIDEO_SAA711X VIDEO_STK1160 VIDEO_TUNER VIDEO_TVEEPROM VIDEO_USBTV VIDEO_V4L2_I2C VIDEO_V4L2_SUBDEV_API VIDEO_V4L2_TPG VIDEO_VICODEC VIDEO_VIM2M VIDEO_VIMC VIDEO_VIVID VIDEO_VIVID_CEC VIDEO_WM8775 VIPERBOARD_ADC VIRTIO_BALLOON VIRTIO_DMA_SHARED_BUFFER VIRTIO_MEM VIRTIO_MMIO VIRTIO_MMIO_CMDLINE_DEVICES VIRTIO_PMEM VIRTIO_VDPA VIRTIO_VSOCKETS VIRTIO_VSOCKETS_COMMON VIRT_WIFI VLAN_8021Q VLAN_8021Q_GVRP VLAN_8021Q_MVRP VMAP_PFN VMWARE_VMCI VMXNET3 VP_VDPA VSOCKETS VSOCKETS_DIAG VSOCKETS_LOOPBACK VSOCKMON VT_HW_CONSOLE_BINDING VXFS_FS WANT_DEV_COREDUMP WEXT_CORE WEXT_PRIV WEXT_PROC WIREGUARD WIRELESS WIRELESS_EXT WLAN WLAN_VENDOR_ADMTEK WLAN_VENDOR_PURELIFI WLAN_VENDOR_SILABS X86_SGX X86_SGX_KVM X86_USER_SHADOW_STACK X86_X2APIC X86_X32_ABI XARRAY_MULTI XDP_SOCKETS XDP_SOCKETS_DIAG XFRM_ESPINTCP XFRM_INTERFACE XFRM_IPCOMP XFRM_MIGRATE XFRM_OFFLOAD XFRM_STATISTICS XFRM_SUB_POLICY XFRM_USER_COMPAT XFS_FS XFS_POSIX_ACL XFS_QUOTA XFS_RT XOR_BLOCKS YENTA YENTA_ENE_TUNE YENTA_O2 YENTA_RICOH YENTA_TI YENTA_TOSHIBA ZBUD ZEROPLUS_FF ZLIB_DEFLATE ZONEFS_FS ZPOOL ZRAM ZRAM_DEF_COMP_LZORLE ZSMALLOC ZSTD_COMPRESS ZSWAP ZSWAP_COMPRESSOR_DEFAULT_LZO ZSWAP_DEFAULT_ON ZSWAP_ZPOOL_DEFAULT_ZBUD] disabling configs for [LEAK UBSAN KASAN LOCKDEP ATOMIC_SLEEP HANG], they are not needed picked [v6.5 v6.4 v6.3 v6.1 v5.19 v5.17 v5.15 v5.13 v5.10 v5.7 v5.4 v5.1 v4.19] out of 28 release tags testing release v6.5 testing commit 2dde18cd1d8fac735875f2e4987f11817cc0bc2c gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: dcae48d9444298af0bd4e6ccac813779f44e7205fc1e834596bc01dc82994abe all runs: OK false negative chance: 0.000 # git bisect start a59addacf899b1b21a7b7449a1c52c98704c2472 2dde18cd1d8fac735875f2e4987f11817cc0bc2c Bisecting: 6551 revisions left to test after this (roughly 13 steps) [4fb0dacb78c6a041bbd38ddd998df806af5c2c69] Merge tag 'sound-6.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound testing commit 4fb0dacb78c6a041bbd38ddd998df806af5c2c69 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: c6ab38a2e063e71053aa816c8d87596f0d986ae2a2221d92d58bcaae869a4e08 all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] # git bisect bad 4fb0dacb78c6a041bbd38ddd998df806af5c2c69 Bisecting: 3156 revisions left to test after this (roughly 12 steps) [bd6c11bc43c496cddfc6cf603b5d45365606dbd5] Merge tag 'net-next-6.6' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next testing commit bd6c11bc43c496cddfc6cf603b5d45365606dbd5 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 958ad2c7333de075c7b8e47bb73e00e09645ea4a7a9998ea14f14b9b1b659cbf all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] # git bisect bad bd6c11bc43c496cddfc6cf603b5d45365606dbd5 Bisecting: 1988 revisions left to test after this (roughly 11 steps) [6c9cfb853063f317b2953c5e852b6bac1eb0cade] net: ethernet: mtk_wed: minor change in wed_{tx,rx}info_show testing commit 6c9cfb853063f317b2953c5e852b6bac1eb0cade gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 05092837d5ec51a4a17d3134c9079fcc8e8a988b75d9c0926d83400646d9adf0 all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] # git bisect bad 6c9cfb853063f317b2953c5e852b6bac1eb0cade Bisecting: 980 revisions left to test after this (roughly 10 steps) [81083076a007d3af3f2216ad9be1374de0687d49] Merge tag 'wireless-next-2023-08-04' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next testing commit 81083076a007d3af3f2216ad9be1374de0687d49 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ab19cc0adf17486f30a16af9d4f43bce5a79a3f6c0067c1cd060a5b21932d548 all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] # git bisect bad 81083076a007d3af3f2216ad9be1374de0687d49 Bisecting: 492 revisions left to test after this (roughly 9 steps) [111d5c4797c0f0314de8ef68b50259798ef3a127] Merge tag 'mt76-for-kvalo-2023-07-31' of https://github.com/nbd168/wireless testing commit 111d5c4797c0f0314de8ef68b50259798ef3a127 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8ae87170ad0c45aace9ccf326aa7e17edab9f96d0720277f437a0c94bcbcaae3 all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] # git bisect bad 111d5c4797c0f0314de8ef68b50259798ef3a127 Bisecting: 254 revisions left to test after this (roughly 8 steps) [97083c21c5d3a1ab5e61a96bc6e76a62f7f23576] Merge tag 'linux-can-next-for-6.6-20230719' of git://git.kernel.org/pub/scm/linux/kernel/git/mkl/linux-can-next testing commit 97083c21c5d3a1ab5e61a96bc6e76a62f7f23576 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a4368f2c902943f38ff404f22df444b778d0512bd9cb09029556c8695d432d3c all runs: OK false negative chance: 0.000 # git bisect good 97083c21c5d3a1ab5e61a96bc6e76a62f7f23576 Bisecting: 127 revisions left to test after this (roughly 7 steps) [634fcbcaa4062db39aeb5ac6ed1bc1feb8dd5216] wifi: rtw88: delete timer and free skb queue when unloading testing commit 634fcbcaa4062db39aeb5ac6ed1bc1feb8dd5216 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 435df13436eac5ae9911c13a4b40763c906406b3979b0bd127644eec2ced5416 all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] # git bisect bad 634fcbcaa4062db39aeb5ac6ed1bc1feb8dd5216 Bisecting: 63 revisions left to test after this (roughly 6 steps) [03b123debcbc8db987bda17ed8412cc011064c22] tcp: tcp_enter_quickack_mode() should be static testing commit 03b123debcbc8db987bda17ed8412cc011064c22 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: dc2ca07cda41914cb6f208f253a677a98ce2183e16fc206cec2c02715c3834e8 all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] # git bisect bad 03b123debcbc8db987bda17ed8412cc011064c22 Bisecting: 31 revisions left to test after this (roughly 5 steps) [17f1034dd76d7465d4c0948c5280c6fc64ee0542] selftests/xsk: transmit and receive multi-buffer packets testing commit 17f1034dd76d7465d4c0948c5280c6fc64ee0542 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 84872438a4b5c9d6393c9b5a74d63506e073fc693c51640b6cd3d63901c15ebc all runs: OK false negative chance: 0.000 # git bisect good 17f1034dd76d7465d4c0948c5280c6fc64ee0542 Bisecting: 15 revisions left to test after this (roughly 4 steps) [24cc7564e0de23b60eb76eb71f5c9a984c6b63b4] Merge branch 'bpf-link-support-for-tc-bpf-programs' testing commit 24cc7564e0de23b60eb76eb71f5c9a984c6b63b4 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: e68464c529d62cbcec4f33d32a9daddae958b27bcda7bba0d10456cd194077ec all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] # git bisect bad 24cc7564e0de23b60eb76eb71f5c9a984c6b63b4 Bisecting: 7 revisions left to test after this (roughly 3 steps) [053c8e1f235dc3f69d13375b32f4209228e1cb96] bpf: Add generic attach/detach/query API for multi-progs testing commit 053c8e1f235dc3f69d13375b32f4209228e1cb96 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: fd5cc1d0af6de37bf2074120f279f26f425cd18a5bf22ec3be6649ce2720e52c all runs: OK false negative chance: 0.000 # git bisect good 053c8e1f235dc3f69d13375b32f4209228e1cb96 Bisecting: 3 revisions left to test after this (roughly 2 steps) [4e9c2d9af5612a6992260b3c7d3047c40265861e] libbpf: Add helper macro to clear opts structs testing commit 4e9c2d9af5612a6992260b3c7d3047c40265861e gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 7e71837e5be34a41923d2156ef1e7c6362c252c6c7d0aaae8478ff29a588d53c all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] # git bisect bad 4e9c2d9af5612a6992260b3c7d3047c40265861e Bisecting: 1 revision left to test after this (roughly 1 step) [fe20ce3a512649b7f883e15dfc01eb29bfcf2168] libbpf: Add opts-based attach/detach/query API for tcx testing commit fe20ce3a512649b7f883e15dfc01eb29bfcf2168 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 21b09c3f1b5e235b3b29fce11f3add1a84230233adefe81185a62b4768b7702d all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] # git bisect bad fe20ce3a512649b7f883e15dfc01eb29bfcf2168 Bisecting: 0 revisions left to test after this (roughly 0 steps) [e420bed025071a623d2720a92bc2245c84757ecb] bpf: Add fd-based tcx multi-prog infra with link support testing commit e420bed025071a623d2720a92bc2245c84757ecb gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: dd94fb5878a60d21a240790dfcab1905905d915cf4b0cc0f855250a6a70f5101 all runs: crashed: WARNING in bpf_mprog_pos_before representative crash: WARNING in bpf_mprog_pos_before, types: [WARNING] # git bisect bad e420bed025071a623d2720a92bc2245c84757ecb e420bed025071a623d2720a92bc2245c84757ecb is the first bad commit commit e420bed025071a623d2720a92bc2245c84757ecb Author: Daniel Borkmann Date: Wed Jul 19 16:08:52 2023 +0200 bpf: Add fd-based tcx multi-prog infra with link support This work refactors and adds a lightweight extension ("tcx") to the tc BPF ingress and egress data path side for allowing BPF program management based on fds via bpf() syscall through the newly added generic multi-prog API. The main goal behind this work which we also presented at LPC [0] last year and a recent update at LSF/MM/BPF this year [3] is to support long-awaited BPF link functionality for tc BPF programs, which allows for a model of safe ownership and program detachment. Given the rise in tc BPF users in cloud native environments, this becomes necessary to avoid hard to debug incidents either through stale leftover programs or 3rd party applications accidentally stepping on each others toes. As a recap, a BPF link represents the attachment of a BPF program to a BPF hook point. The BPF link holds a single reference to keep BPF program alive. Moreover, hook points do not reference a BPF link, only the application's fd or pinning does. A BPF link holds meta-data specific to attachment and implements operations for link creation, (atomic) BPF program update, detachment and introspection. The motivation for BPF links for tc BPF programs is multi-fold, for example: - From Meta: "It's especially important for applications that are deployed fleet-wide and that don't "control" hosts they are deployed to. If such application crashes and no one notices and does anything about that, BPF program will keep running draining resources or even just, say, dropping packets. We at FB had outages due to such permanent BPF attachment semantics. With fd-based BPF link we are getting a framework, which allows safe, auto-detachable behavior by default, unless application explicitly opts in by pinning the BPF link." [1] - From Cilium-side the tc BPF programs we attach to host-facing veth devices and phys devices build the core datapath for Kubernetes Pods, and they implement forwarding, load-balancing, policy, EDT-management, etc, within BPF. Currently there is no concept of 'safe' ownership, e.g. we've recently experienced hard-to-debug issues in a user's staging environment where another Kubernetes application using tc BPF attached to the same prio/handle of cls_bpf, accidentally wiping all Cilium-based BPF programs from underneath it. The goal is to establish a clear/safe ownership model via links which cannot accidentally be overridden. [0,2] BPF links for tc can co-exist with non-link attachments, and the semantics are in line also with XDP links: BPF links cannot replace other BPF links, BPF links cannot replace non-BPF links, non-BPF links cannot replace BPF links and lastly only non-BPF links can replace non-BPF links. In case of Cilium, this would solve mentioned issue of safe ownership model as 3rd party applications would not be able to accidentally wipe Cilium programs, even if they are not BPF link aware. Earlier attempts [4] have tried to integrate BPF links into core tc machinery to solve cls_bpf, which has been intrusive to the generic tc kernel API with extensions only specific to cls_bpf and suboptimal/complex since cls_bpf could be wiped from the qdisc also. Locking a tc BPF program in place this way, is getting into layering hacks given the two object models are vastly different. We instead implemented the tcx (tc 'express') layer which is an fd-based tc BPF attach API, so that the BPF link implementation blends in naturally similar to other link types which are fd-based and without the need for changing core tc internal APIs. BPF programs for tc can then be successively migrated from classic cls_bpf to the new tc BPF link without needing to change the program's source code, just the BPF loader mechanics for attaching is sufficient. For the current tc framework, there is no change in behavior with this change and neither does this change touch on tc core kernel APIs. The gist of this patch is that the ingress and egress hook have a lightweight, qdisc-less extension for BPF to attach its tc BPF programs, in other words, a minimal entry point for tc BPF. The name tcx has been suggested from discussion of earlier revisions of this work as a good fit, and to more easily differ between the classic cls_bpf attachment and the fd-based one. For the ingress and egress tcx points, the device holds a cache-friendly array with program pointers which is separated from control plane (slow-path) data. Earlier versions of this work used priority to determine ordering and expression of dependencies similar as with classic tc, but it was challenged that for something more future-proof a better user experience is required. Hence this resulted in the design and development of the generic attach/detach/query API for multi-progs. See prior patch with its discussion on the API design. tcx is the first user and later we plan to integrate also others, for example, one candidate is multi-prog support for XDP which would benefit and have the same 'look and feel' from API perspective. The goal with tcx is to have maximum compatibility to existing tc BPF programs, so they don't need to be rewritten specifically. Compatibility to call into classic tcf_classify() is also provided in order to allow successive migration or both to cleanly co-exist where needed given its all one logical tc layer and the tcx plus classic tc cls/act build one logical overall processing pipeline. tcx supports the simplified return codes TCX_NEXT which is non-terminating (go to next program) and terminating ones with TCX_PASS, TCX_DROP, TCX_REDIRECT. The fd-based API is behind a static key, so that when unused the code is also not entered. The struct tcx_entry's program array is currently static, but could be made dynamic if necessary at a point in future. The a/b pair swap design has been chosen so that for detachment there are no allocations which otherwise could fail. The work has been tested with tc-testing selftest suite which all passes, as well as the tc BPF tests from the BPF CI, and also with Cilium's L4LB. Thanks also to Nikolay Aleksandrov and Martin Lau for in-depth early reviews of this work. [0] https://lpc.events/event/16/contributions/1353/ [1] https://lore.kernel.org/bpf/CAEf4BzbokCJN33Nw_kg82sO=xppXnKWEncGTWCTB9vGCmLB6pw@mail.gmail.com [2] https://colocatedeventseu2023.sched.com/event/1Jo6O/tales-from-an-ebpf-programs-murder-mystery-hemanth-malla-guillaume-fournier-datadog [3] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf [4] https://lore.kernel.org/bpf/20210604063116.234316-1-memxor@gmail.com Signed-off-by: Daniel Borkmann Acked-by: Jakub Kicinski Link: https://lore.kernel.org/r/20230719140858.13224-3-daniel@iogearbox.net Signed-off-by: Alexei Starovoitov MAINTAINERS | 4 +- include/linux/bpf_mprog.h | 9 ++ include/linux/netdevice.h | 15 +- include/linux/skbuff.h | 4 +- include/net/sch_generic.h | 2 +- include/net/tcx.h | 206 ++++++++++++++++++++++++ include/uapi/linux/bpf.h | 34 +++- kernel/bpf/Kconfig | 1 + kernel/bpf/Makefile | 1 + kernel/bpf/syscall.c | 82 ++++++++-- kernel/bpf/tcx.c | 348 +++++++++++++++++++++++++++++++++++++++++ net/Kconfig | 5 + net/core/dev.c | 265 +++++++++++++++++++------------ net/core/filter.c | 4 +- net/sched/Kconfig | 4 +- net/sched/sch_ingress.c | 61 +++++++- tools/include/uapi/linux/bpf.h | 34 +++- 17 files changed, 935 insertions(+), 144 deletions(-) create mode 100644 include/net/tcx.h create mode 100644 kernel/bpf/tcx.c accumulated error probability: 0.00 culprit signature: dd94fb5878a60d21a240790dfcab1905905d915cf4b0cc0f855250a6a70f5101 parent signature: fd5cc1d0af6de37bf2074120f279f26f425cd18a5bf22ec3be6649ce2720e52c revisions tested: 22, total time: 8h44m3.35815622s (build: 5h29m50.624421431s, test: 2h35m16.352517015s) first bad commit: e420bed025071a623d2720a92bc2245c84757ecb bpf: Add fd-based tcx multi-prog infra with link support recipients (to): ["ast@kernel.org" "daniel@iogearbox.net" "kuba@kernel.org"] recipients (cc): [] crash: WARNING in bpf_mprog_pos_before ------------[ cut here ]------------ WARNING: CPU: 1 PID: 2924 at include/linux/bpf_mprog.h:198 bpf_mprog_pos_before+0x44/0x60 kernel/bpf/mprog.c:204 Modules linked in: CPU: 1 PID: 2924 Comm: syz-executor.0 Not tainted 6.5.0-rc1-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/04/2023 RIP: 0010:bpf_mprog_total include/linux/bpf_mprog.h:198 [inline] RIP: 0010:bpf_mprog_pos_before+0x44/0x60 kernel/bpf/mprog.c:200 Code: f9 3f 89 c2 7f 24 39 c1 7e 26 48 8b 14 d7 49 39 d0 75 e8 48 8b 56 08 48 85 d2 74 0a 49 3b 94 c1 10 04 00 00 75 d5 83 e8 01 c3 <0f> 0b 39 c1 7f da 49 83 f8 01 b8 fe ff ff ff 83 d0 00 c3 66 0f 1f RSP: 0018:ffffc90000eb3d30 EFLAGS: 00010212 RAX: 0000000000000000 RBX: ffffc90000ea1000 RCX: 0000000000000040 RDX: 0000000000000000 RSI: ffffc90000eb3d70 RDI: ffff888112d6d808 RBP: ffff88811a152000 R08: 0000000000000000 R09: ffff888112d6d808 R10: 0000000000000000 R11: 0000000000000000 R12: ffff888112d6d808 R13: 0000000000000008 R14: 0000000000000000 R15: 0000000000000000 FS: 00007f1453c5d6c0(0000) GS:ffff888237d00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000110341000 CR4: 00000000003506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: bpf_mprog_attach+0x32a/0x570 kernel/bpf/mprog.c:258 tcx_prog_attach+0xb8/0x2f0 kernel/bpf/tcx.c:39 bpf_prog_attach kernel/bpf/syscall.c:3820 [inline] __sys_bpf+0x28b/0x2600 kernel/bpf/syscall.c:5354 __do_sys_bpf kernel/bpf/syscall.c:5449 [inline] __se_sys_bpf kernel/bpf/syscall.c:5447 [inline] __x64_sys_bpf+0x19/0x20 kernel/bpf/syscall.c:5447 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x38/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd RIP: 0033:0x7f14540daae9 Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f1453c5d0c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 00007f14541f9f80 RCX: 00007f14540daae9 RDX: 0000000000000020 RSI: 0000000020000080 RDI: 0000000000000008 RBP: 00007f145412647a R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 000000000000000b R14: 00007f14541f9f80 R15: 00007fff808a4b18