ci starts bisection 2023-07-22 00:53:03.049025219 +0000 UTC m=+10287.337460341 bisecting cause commit starting from 03b123debcbc8db987bda17ed8412cc011064c22 building syzkaller on 7b630fdbfd1d53f913e0aff0dfa8ebfbaf86652b ensuring issue is reproducible on original commit 03b123debcbc8db987bda17ed8412cc011064c22 testing commit 03b123debcbc8db987bda17ed8412cc011064c22 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 7e731f17fef87952714fd4b754078ac6c4ba1d4036f72464db8c17d3546f509c all runs: crashed: WARNING: ODEBUG bug in ingress_destroy representative crash: WARNING: ODEBUG bug in ingress_destroy, types: [WARNING] check whether we can drop unnecessary instrumentation disabling configs for [ATOMIC_SLEEP HANG LEAK UBSAN KASAN LOCKDEP], they are not needed testing commit 03b123debcbc8db987bda17ed8412cc011064c22 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: bb87317522acbdcb113d462d8ceca9c2f104641f6cd6421467b06034ab8d5836 all runs: crashed: WARNING: ODEBUG bug in ingress_destroy representative crash: WARNING: ODEBUG bug in ingress_destroy, types: [WARNING] the bug reproduces without the instrumentation disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed kconfig minimization: base=3883 full=7653 leaves diff=1998 split chunks (needed=false): <1998> split chunk #0 of len 1998 into 5 parts testing without sub-chunk 1/5 testing commit 03b123debcbc8db987bda17ed8412cc011064c22 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9d5a2ac11f0b993d7118ca618e77aefc3920116cbf7f8425bd68833b3d7052dd all runs: crashed: WARNING: ODEBUG bug in ingress_destroy representative crash: WARNING: ODEBUG bug in ingress_destroy, types: [WARNING] the chunk can be dropped testing without sub-chunk 2/5 testing commit 03b123debcbc8db987bda17ed8412cc011064c22 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 32f7d6088eb09b9f69cf97122c03f3c4dd414f9c9f287b87a7b3a8a8e5409820 all runs: crashed: WARNING: ODEBUG bug in ingress_destroy representative crash: WARNING: ODEBUG bug in ingress_destroy, types: [WARNING] the chunk can be dropped testing without sub-chunk 3/5 testing commit 03b123debcbc8db987bda17ed8412cc011064c22 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 046dffbadd106c2a55d9cbfcacb5a0b7d3f5a6a56d69b8d559c074b20fcaeee2 all runs: OK false negative chance: 0.000 testing without sub-chunk 4/5 testing commit 03b123debcbc8db987bda17ed8412cc011064c22 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d145f3c4378551f8dc3fd59d7c26ec5f89c09953acf834eba9917986801ab6a7 all runs: crashed: WARNING: ODEBUG bug in ingress_destroy representative crash: WARNING: ODEBUG bug in ingress_destroy, types: [WARNING] the chunk can be dropped testing without sub-chunk 5/5 testing commit 03b123debcbc8db987bda17ed8412cc011064c22 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 85e18e254ee3b0961f3fde1ac258761b3db0f6a8c9a49a3754c43c4254309637 all runs: crashed: WARNING: ODEBUG bug in ingress_destroy representative crash: WARNING: ODEBUG bug in ingress_destroy, types: [WARNING] the chunk can be dropped minimized to 400 configs; suspects: [AX25 BRIDGE BRIDGE_NETFILTER CAN CFG80211 CHECKPOINT_RESTORE DVB_CORE FB FSCACHE HAMRADIO HSR INFINIBAND INFINIBAND_ADDR_TRANS INFINIBAND_USER_ACCESS INPUT_JOYSTICK INPUT_MOUSE IP6_NF_RAW IPV6_MULTIPLE_TABLES IP_NF_RAW IP_SET IP_VS IP_VS_PE_SIP IP_VS_PROTO_AH IP_VS_PROTO_AH_ESP IP_VS_PROTO_ESP IP_VS_PROTO_SCTP IP_VS_PROTO_UDP IP_VS_RR IP_VS_SED IP_VS_SH IP_VS_TWOS IP_VS_WLC IP_VS_WRR IRQ_BYPASS_MANAGER IRQ_POLL IR_IGORPLUGUSB IR_IGUANA IR_IMON IR_MCEUSB IR_REDRAT3 IR_STREAMZAP IR_TTUSBIR ISDN ISDN_CAPI_MIDDLEWARE JFFS2_CMODE_PRIORITY JFFS2_COMPRESSION_OPTIONS JFFS2_FS JFFS2_FS_POSIX_ACL JFFS2_FS_SECURITY JFFS2_FS_WRITEBUFFER JFFS2_FS_XATTR JFFS2_LZO JFFS2_RTIME JFFS2_RUBIN JFFS2_SUMMARY JFFS2_ZLIB JFS_DEBUG JFS_FS JFS_POSIX_ACL JFS_SECURITY JOYSTICK_IFORCE JOYSTICK_IFORCE_USB JOYSTICK_XPAD JOYSTICK_XPAD_FF JOYSTICK_XPAD_LEDS KARMA_PARTITION KCOV KCOV_ENABLE_COMPARISONS KCOV_INSTRUMENT_ALL KEYS_REQUEST_CACHE KEY_DH_OPERATIONS KEY_NOTIFICATIONS KSM KVM KVM_AMD KVM_ASYNC_PF KVM_COMPAT KVM_GENERIC_DIRTYLOG_READ_PROTECT KVM_GENERIC_HARDWARE_ENABLING KVM_MMIO KVM_VFIO KVM_XEN KVM_XFER_TO_GUEST_WORK L2TP L2TP_ETH L2TP_IP L2TP_V3 LAPB LAPBETHER LDM_PARTITION LEDS_TRIGGER_AUDIO LEGACY_PTYS LIBCRC32C LIBNVDIMM LINEAR_RANGES LLC LLC2 LOGIG940_FF LOGIRUMBLEPAD2_FF LOGO LOGO_LINUX_MONO LOGO_LINUX_VGA16 LPC_ICH LRU_GEN LRU_GEN_ENABLED LWTUNNEL LWTUNNEL_BPF LZ4HC_COMPRESS LZ4_COMPRESS MAC80211 MAC80211_DEBUGFS MAC80211_HAS_RC MAC80211_HWSIM MAC80211_MESH MAC80211_RC_DEFAULT_MINSTREL MAC80211_RC_MINSTREL MACSEC MACVLAN MACVTAP MAC_PARTITION MAPPING_DIRTY_HELPERS MD_LINEAR MD_MULTIPATH MD_RAID0 MD_RAID1 MD_RAID10 MD_RAID456 MEDIA_ANALOG_TV_SUPPORT MEDIA_ATTACH MEDIA_CONTROLLER MEDIA_CONTROLLER_DVB MEDIA_CONTROLLER_REQUEST_API MEDIA_DIGITAL_TV_SUPPORT MEDIA_RADIO_SUPPORT MEDIA_SDR_SUPPORT MEDIA_SUPPORT MEDIA_SUPPORT_FILTER MEDIA_TUNER MEDIA_TUNER_MSI001 MEMORY_BALLOON MEMORY_HOTPLUG MEMORY_HOTPLUG_DEFAULT_ONLINE MEMORY_ISOLATION MEMREGION MEMSTICK MEMSTICK_REALTEK_USB MEM_SOFT_DIRTY MFD_CORE MFD_SYSCON MHI_BUS MHI_WWAN_CTRL MHP_MEMMAP_ON_MEMORY MICROCHIP_PHY MINIX_FS MINIX_SUBPARTITION MISC_RTSX MISC_RTSX_USB MISDN MISDN_DSP MISDN_HFCUSB MISDN_L1OIP MKISS MLX4_CORE MLX4_INFINIBAND MMC MMC_REALTEK_USB MMC_USHC MMC_VUB300 MMU_NOTIFIER MODULE_SRCVERSION_ALL MODVERSIONS MOST MOUSE_APPLETOUCH MOUSE_BCM5974 MOUSE_PS2 MOUSE_PS2_ALPS MOUSE_PS2_BYD MOUSE_PS2_CYPRESS MOUSE_PS2_FOCALTECH MOUSE_PS2_LIFEBOOK MOUSE_PS2_LOGIPS2PP MOUSE_PS2_SMBUS MOUSE_PS2_SYNAPTICS MOUSE_PS2_SYNAPTICS_SMBUS MOUSE_PS2_TRACKPOINT MOUSE_SYNAPTICS_USB MPLS MPLS_IPTUNNEL MPLS_ROUTING MPTCP MPTCP_IPV6 MRP MTD MTD_BLKDEVS MTD_BLOCK MTD_BLOCK2MTD MTD_CFI_I1 MTD_CFI_I2 MTD_MAP_BANK_WIDTH_1 MTD_MAP_BANK_WIDTH_2 MTD_MAP_BANK_WIDTH_4 MTD_MTDRAM MTD_PHRAM MTD_SLRAM MUSB_PIO_ONLY ND_BTT ND_CLAIM ND_PFN NETDEVSIM NETFILTER_ADVANCED NETFILTER_BPF_LINK NETFILTER_FAMILY_ARP NETFILTER_FAMILY_BRIDGE NETFILTER_NETLINK_ACCT NETFILTER_NETLINK_GLUE_CT NETFILTER_NETLINK_OSF NETFILTER_NETLINK_QUEUE NETFILTER_SYNPROXY NETFILTER_XTABLES_COMPAT NETFILTER_XT_CONNMARK NETFILTER_XT_MATCH_BPF NETFILTER_XT_MATCH_CGROUP NETFILTER_XT_MATCH_CLUSTER NETFILTER_XT_MATCH_COMMENT NETFILTER_XT_MATCH_CONNBYTES NETFILTER_XT_MATCH_CONNLABEL NETFILTER_XT_MATCH_CONNLIMIT NETFILTER_XT_MATCH_CONNMARK NETFILTER_XT_MATCH_CPU NETFILTER_XT_MATCH_DCCP NETFILTER_XT_MATCH_DEVGROUP NETFILTER_XT_MATCH_DSCP NETFILTER_XT_MATCH_ECN NETFILTER_XT_MATCH_ESP NETFILTER_XT_MATCH_HASHLIMIT NETFILTER_XT_MATCH_HELPER NETFILTER_XT_MATCH_HL NETFILTER_XT_MATCH_IPCOMP NETFILTER_XT_MATCH_IPRANGE NETFILTER_XT_MATCH_IPVS NETFILTER_XT_MATCH_L2TP NETFILTER_XT_MATCH_LENGTH NETFILTER_XT_MATCH_LIMIT NETFILTER_XT_MATCH_MAC NETFILTER_XT_MATCH_MARK NETFILTER_XT_MATCH_MULTIPORT NETFILTER_XT_MATCH_NFACCT NETFILTER_XT_MATCH_OSF NETFILTER_XT_MATCH_OWNER NETFILTER_XT_MATCH_PHYSDEV NETFILTER_XT_MATCH_PKTTYPE NETFILTER_XT_MATCH_QUOTA NETFILTER_XT_MATCH_RATEEST NETFILTER_XT_MATCH_REALM NETFILTER_XT_MATCH_RECENT NETFILTER_XT_MATCH_SCTP NETFILTER_XT_MATCH_SOCKET NETFILTER_XT_MATCH_STATISTIC NETFILTER_XT_MATCH_STRING NETFILTER_XT_MATCH_TCPMSS NETFILTER_XT_MATCH_TIME NETFILTER_XT_MATCH_U32 NETFILTER_XT_SET NETFILTER_XT_TARGET_AUDIT NETFILTER_XT_TARGET_CHECKSUM NETFILTER_XT_TARGET_CLASSIFY NETFILTER_XT_TARGET_CONNMARK NETFILTER_XT_TARGET_CT NETFILTER_XT_TARGET_DSCP NETFILTER_XT_TARGET_HL NETFILTER_XT_TARGET_HMARK NETFILTER_XT_TARGET_IDLETIMER NETFILTER_XT_TARGET_LED NETFILTER_XT_TARGET_MARK NETFILTER_XT_TARGET_NETMAP NETFILTER_XT_TARGET_NFQUEUE NETFILTER_XT_TARGET_NOTRACK NETFILTER_XT_TARGET_RATEEST NETFILTER_XT_TARGET_REDIRECT NETFILTER_XT_TARGET_TCPOPTSTRIP NETFILTER_XT_TARGET_TEE NETFILTER_XT_TARGET_TPROXY NETFILTER_XT_TARGET_TRACE NETLABEL NETLINK_DIAG NETROM NET_9P_RDMA NET_ACT_BPF NET_ACT_CONNMARK NET_ACT_CSUM NET_ACT_CT NET_ACT_CTINFO NET_ACT_GATE NET_ACT_IFE NET_ACT_IPT NET_ACT_MPLS NET_ACT_NAT NET_ACT_PEDIT NET_ACT_POLICE NET_ACT_SAMPLE NET_ACT_SIMP NET_ACT_SKBEDIT NET_ACT_SKBMOD NET_ACT_TUNNEL_KEY NET_ACT_VLAN NET_CLS_BASIC NET_CLS_BPF NET_CLS_FLOW NET_CLS_FLOWER NET_CLS_FW NET_CLS_MATCHALL NET_CLS_ROUTE4 NET_DEVLINK NET_DROP_MONITOR NET_DSA NET_DSA_TAG_BRCM NET_DSA_TAG_BRCM_COMMON NET_DSA_TAG_BRCM_PREPEND NET_DSA_TAG_MTK NET_DSA_TAG_QCA NET_DSA_TAG_RTL4_A NET_EMATCH_CANID NET_EMATCH_CMP NET_EMATCH_IPSET NET_EMATCH_IPT NET_EMATCH_META NET_EMATCH_NBYTE NET_EMATCH_TEXT NET_EMATCH_U32 NET_FC NET_FOU NET_FOU_IP_TUNNELS NET_IFE NET_IFE_SKBMARK NET_IFE_SKBPRIO NET_IFE_SKBTCINDEX NET_IPGRE NET_IPGRE_BROADCAST NET_IPGRE_DEMUX NET_IPIP NET_IPVTI NET_KEY NET_KEY_MIGRATE NET_L3_MASTER_DEV NET_MPLS_GSO NET_NCSI NET_NSH NET_REDIRECT NET_SCH_CAKE NET_SCH_CBS NET_SCH_CHOKE NET_SCH_CODEL NET_SCH_DRR NET_SCH_ETF NET_SCH_ETS NET_SCH_FQ NET_SCH_FQ_CODEL NET_SCH_FQ_PIE NET_SCH_GRED NET_SCH_HFSC NET_SCH_HHF NET_SCH_HTB NET_SCH_INGRESS NET_SCH_MQPRIO NET_SCH_MQPRIO_LIB NET_SCH_MULTIQ NET_SCH_NETEM NET_SCH_PIE NET_SCH_PLUG NET_SCH_PRIO NET_SCH_QFQ NET_SCH_RED NET_SCH_SFB NET_SCH_SFQ NET_SCH_SKBPRIO NET_SCH_TAPRIO NET_SCH_TBF NET_SCH_TEQL NET_SOCK_MSG NET_SWITCHDEV NET_TC_SKB_EXT NET_TEAM NET_TEAM_MODE_ACTIVEBACKUP NET_TEAM_MODE_BROADCAST NET_TEAM_MODE_LOADBALANCE NET_TEAM_MODE_RANDOM NET_TEAM_MODE_ROUNDROBIN NET_UDP_TUNNEL NET_VRF NET_XGRESS NFC NFC_DIGITAL NFC_FDP NFC_HCI NFC_MRVL NFC_MRVL_USB NFC_NCI NFC_NCI_UART NFC_PN533 NFC_PN533_USB NFC_PORT100 NFC_SHDLC NFC_SIM NFC_VIRTUAL_NCI NFSD NFSD_BLOCKLAYOUT NFSD_FLEXFILELAYOUT NFSD_PNFS NFSD_SCSILAYOUT NFSD_V3_ACL NFSD_V4 NFSD_V4_2_INTER_SSC NFSD_V4_SECURITY_LABEL NFS_FSCACHE NFS_V4_1 NFS_V4_2 NFS_V4_2_READ_PLUS NFS_V4_2_SSC_HELPER NFS_V4_SECURITY_LABEL NFT_BRIDGE_META NFT_BRIDGE_REJECT NFT_COMPAT NFT_CONNLIMIT NFT_CT NFT_DUP_IPV4 NFT_DUP_IPV6 NFT_DUP_NETDEV NFT_FIB NFT_FIB_INET NFT_FIB_IPV4 NFT_FIB_IPV6 NFT_FIB_NETDEV NFT_FLOW_OFFLOAD NFT_HASH NFT_LIMIT NFT_LOG NFT_MASQ NFT_NAT NFT_NUMGEN NFT_OSF NFT_QUEUE NFT_QUOTA NFT_REDIR NFT_REJECT NFT_REJECT_INET NFT_REJECT_IPV4 NFT_REJECT_IPV6 NFT_REJECT_NETDEV NFT_SOCKET NFT_SYNPROXY NFT_TPROXY NFT_TUNNEL NFT_XFRM NF_CONNTRACK_AMANDA NF_CONNTRACK_BRIDGE NF_CONNTRACK_BROADCAST NF_CONNTRACK_EVENTS NF_CONNTRACK_H323 NF_CONNTRACK_LABELS NF_CONNTRACK_MARK NF_CONNTRACK_NETBIOS_NS NF_CONNTRACK_OVS NF_CONNTRACK_PPTP NF_CONNTRACK_SANE NF_CONNTRACK_SNMP NF_CONNTRACK_TFTP NF_CONNTRACK_TIMEOUT NF_CONNTRACK_TIMESTAMP NF_CONNTRACK_ZONES NF_CT_NETLINK_HELPER NF_CT_NETLINK_TIMEOUT NF_CT_PROTO_DCCP NF_CT_PROTO_GRE NF_CT_PROTO_SCTP NF_CT_PROTO_UDPLITE NF_DUP_IPV4 NF_DUP_IPV6 NF_DUP_NETDEV NF_FLOW_TABLE NF_FLOW_TABLE_INET NF_NAT_AMANDA NF_NAT_H323 NF_NAT_OVS NF_NAT_PPTP NF_NAT_REDIRECT NF_NAT_SNMP_BASIC NF_NAT_TFTP NF_TABLES NF_TABLES_BRIDGE NF_TABLES_INET NF_TABLES_IPV4 NF_TABLES_IPV6 NF_TABLES_NETDEV PARTITION_ADVANCED PSAMPLE RC_CORE RC_DEVICES RFKILL SPI USB_GADGET USB_MUSB_HDRC VIDEO_DEV WAN WATCH_QUEUE WIRELESS WLAN WWAN X25 X86_X32_ABI] disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing release v6.4 testing commit 6995e2de6891c724bfeb2db33d7b87775f913ad1 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: b290db38ef29e2ba8b49625f8f29cba2eed04e30453ca5df5828633e39495b36 all runs: OK false negative chance: 0.000 # git bisect start 03b123debcbc8db987bda17ed8412cc011064c22 6995e2de6891c724bfeb2db33d7b87775f913ad1 Bisecting: 5883 revisions left to test after this (roughly 13 steps) [1b722407a13b7f8658d2e26917791f32805980a2] Merge tag 'drm-next-2023-06-29' of git://anongit.freedesktop.org/drm/drm testing commit 1b722407a13b7f8658d2e26917791f32805980a2 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: f137f73e52b7f9a175837cb9f1ac49492fea3c0c8c07757c22e8a1fa29008a61 all runs: OK false negative chance: 0.000 # git bisect good 1b722407a13b7f8658d2e26917791f32805980a2 Bisecting: 2962 revisions left to test after this (roughly 12 steps) [e3c2b10d6f15640407bef3098accf10faa4ecf1b] Merge tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux testing commit e3c2b10d6f15640407bef3098accf10faa4ecf1b gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: aad878f8f75cf59fc10dc818d72956a6ac0a2e5f423b76f0d930c8b00f60aafb all runs: OK false negative chance: 0.000 # git bisect good e3c2b10d6f15640407bef3098accf10faa4ecf1b Bisecting: 1480 revisions left to test after this (roughly 11 steps) [94c76955e86a5a4f16a1d690b66dcc268c156e6a] Merge tag 'gfs2-v6.4-rc5-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/gfs2/linux-gfs2 testing commit 94c76955e86a5a4f16a1d690b66dcc268c156e6a gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 51bc1a217b1da1cf9b552a54de2be441f783308f6022337811145d87291e341c all runs: OK false negative chance: 0.000 # git bisect good 94c76955e86a5a4f16a1d690b66dcc268c156e6a Bisecting: 764 revisions left to test after this (roughly 10 steps) [94e0d43e51ff8577ad273032bb1cacfd68e9297b] Merge tag 'acpi-6.5-rc1-3' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm testing commit 94e0d43e51ff8577ad273032bb1cacfd68e9297b gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: d69883c9384a2af99e5c708680ee8791907a18d760bf528518aac550b40f80a3 all runs: OK false negative chance: 0.000 # git bisect good 94e0d43e51ff8577ad273032bb1cacfd68e9297b Bisecting: 402 revisions left to test after this (roughly 9 steps) [ebc27aaceeb9c4c988dbf321e849fcb75c6b55fa] Merge tag 'trace-v6.5-rc1-3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace testing commit ebc27aaceeb9c4c988dbf321e849fcb75c6b55fa gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: bc2b36efb040bdf737fc52455608aea8bac9dccd6d413c872656a0d17ccd1621 all runs: OK false negative chance: 0.000 # git bisect good ebc27aaceeb9c4c988dbf321e849fcb75c6b55fa Bisecting: 201 revisions left to test after this (roughly 8 steps) [6f13b49b57f2cdcb9cab3841da8a441c3521d64e] Merge branch 'selftests-mptcp-join-pass-args-in-new-env-vars' testing commit 6f13b49b57f2cdcb9cab3841da8a441c3521d64e gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9751adee3b786134f96745fdea95b394ef08abc792923ae9d1e9cc61c8f0cf52 all runs: OK false negative chance: 0.000 # git bisect good 6f13b49b57f2cdcb9cab3841da8a441c3521d64e Bisecting: 100 revisions left to test after this (roughly 7 steps) [8bb5e82589f0141a990d3fd21d5b79a73a8c6c7b] ip_tunnels: Add nexthop ID field to ip_tunnel_key testing commit 8bb5e82589f0141a990d3fd21d5b79a73a8c6c7b gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 7fec359d44aff9d119b4190d86b431d1438d922294c84df2e5bbaa6be4a287b1 all runs: OK false negative chance: 0.000 # git bisect good 8bb5e82589f0141a990d3fd21d5b79a73a8c6c7b Bisecting: 51 revisions left to test after this (roughly 6 steps) [6f5a630d7c57cd79b1f526a95e757311e32d41e5] bpf, net: Introduce skb_pointer_if_linear(). testing commit 6f5a630d7c57cd79b1f526a95e757311e32d41e5 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: e525f27ab0f9028ff65c467da7c63e0a13ca4dcf44ec54b3a88738b3c8fb9aaa all runs: crashed: WARNING: ODEBUG bug in ingress_destroy representative crash: WARNING: ODEBUG bug in ingress_destroy, types: [WARNING] # git bisect bad 6f5a630d7c57cd79b1f526a95e757311e32d41e5 Bisecting: 24 revisions left to test after this (roughly 5 steps) [24ea50127ecf0efe819c1f6230add27abc6ca9d9] xsk: support mbuf on ZC RX testing commit 24ea50127ecf0efe819c1f6230add27abc6ca9d9 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3e32234b6e52ae2125bddda8e086c8bd5a26d3f06215b69915aac63b7619643f all runs: OK false negative chance: 0.000 # git bisect good 24ea50127ecf0efe819c1f6230add27abc6ca9d9 Bisecting: 12 revisions left to test after this (roughly 4 steps) [807bf4da204938f01ad2e994ba92e82ac486ea2b] selftests/xsk: add test for too many frags testing commit 807bf4da204938f01ad2e994ba92e82ac486ea2b gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a8d91e99df517536beb29624a23218fca06cca6b98e3b4d47eefd7d66aaef1fe all runs: OK false negative chance: 0.000 # git bisect good 807bf4da204938f01ad2e994ba92e82ac486ea2b Bisecting: 6 revisions left to test after this (roughly 3 steps) [55cc3768473e139483be8f17796b50d21788953f] libbpf: Add link-based API for tcx testing commit 55cc3768473e139483be8f17796b50d21788953f gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 74c542a8b7db0aa36cf47b3ffc324f7bf843eb052e4a1ee0268995a31c22d67d all runs: crashed: WARNING: ODEBUG bug in ingress_destroy representative crash: WARNING: ODEBUG bug in ingress_destroy, types: [WARNING] # git bisect bad 55cc3768473e139483be8f17796b50d21788953f Bisecting: 2 revisions left to test after this (roughly 2 steps) [053c8e1f235dc3f69d13375b32f4209228e1cb96] bpf: Add generic attach/detach/query API for multi-progs testing commit 053c8e1f235dc3f69d13375b32f4209228e1cb96 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 53788cea7907e38be7bdb8267bb985531804710e04ae29aed9571ce4b72802c7 all runs: OK false negative chance: 0.000 # git bisect good 053c8e1f235dc3f69d13375b32f4209228e1cb96 Bisecting: 0 revisions left to test after this (roughly 1 step) [fe20ce3a512649b7f883e15dfc01eb29bfcf2168] libbpf: Add opts-based attach/detach/query API for tcx testing commit fe20ce3a512649b7f883e15dfc01eb29bfcf2168 gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 5ed7c20c4a07a07b341daece22726e2c8a311510a52debd5f695e22ddbc5509b all runs: crashed: WARNING: ODEBUG bug in ingress_destroy representative crash: WARNING: ODEBUG bug in ingress_destroy, types: [WARNING] # git bisect bad fe20ce3a512649b7f883e15dfc01eb29bfcf2168 Bisecting: 0 revisions left to test after this (roughly 0 steps) [e420bed025071a623d2720a92bc2245c84757ecb] bpf: Add fd-based tcx multi-prog infra with link support testing commit e420bed025071a623d2720a92bc2245c84757ecb gcc compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 00c3e06d4ef86913dcba41390a7cf91659c7332620641bc23932cdecb55aaccd all runs: crashed: WARNING: ODEBUG bug in ingress_destroy representative crash: WARNING: ODEBUG bug in ingress_destroy, types: [WARNING] # git bisect bad e420bed025071a623d2720a92bc2245c84757ecb e420bed025071a623d2720a92bc2245c84757ecb is the first bad commit commit e420bed025071a623d2720a92bc2245c84757ecb Author: Daniel Borkmann Date: Wed Jul 19 16:08:52 2023 +0200 bpf: Add fd-based tcx multi-prog infra with link support This work refactors and adds a lightweight extension ("tcx") to the tc BPF ingress and egress data path side for allowing BPF program management based on fds via bpf() syscall through the newly added generic multi-prog API. The main goal behind this work which we also presented at LPC [0] last year and a recent update at LSF/MM/BPF this year [3] is to support long-awaited BPF link functionality for tc BPF programs, which allows for a model of safe ownership and program detachment. Given the rise in tc BPF users in cloud native environments, this becomes necessary to avoid hard to debug incidents either through stale leftover programs or 3rd party applications accidentally stepping on each others toes. As a recap, a BPF link represents the attachment of a BPF program to a BPF hook point. The BPF link holds a single reference to keep BPF program alive. Moreover, hook points do not reference a BPF link, only the application's fd or pinning does. A BPF link holds meta-data specific to attachment and implements operations for link creation, (atomic) BPF program update, detachment and introspection. The motivation for BPF links for tc BPF programs is multi-fold, for example: - From Meta: "It's especially important for applications that are deployed fleet-wide and that don't "control" hosts they are deployed to. If such application crashes and no one notices and does anything about that, BPF program will keep running draining resources or even just, say, dropping packets. We at FB had outages due to such permanent BPF attachment semantics. With fd-based BPF link we are getting a framework, which allows safe, auto-detachable behavior by default, unless application explicitly opts in by pinning the BPF link." [1] - From Cilium-side the tc BPF programs we attach to host-facing veth devices and phys devices build the core datapath for Kubernetes Pods, and they implement forwarding, load-balancing, policy, EDT-management, etc, within BPF. Currently there is no concept of 'safe' ownership, e.g. we've recently experienced hard-to-debug issues in a user's staging environment where another Kubernetes application using tc BPF attached to the same prio/handle of cls_bpf, accidentally wiping all Cilium-based BPF programs from underneath it. The goal is to establish a clear/safe ownership model via links which cannot accidentally be overridden. [0,2] BPF links for tc can co-exist with non-link attachments, and the semantics are in line also with XDP links: BPF links cannot replace other BPF links, BPF links cannot replace non-BPF links, non-BPF links cannot replace BPF links and lastly only non-BPF links can replace non-BPF links. In case of Cilium, this would solve mentioned issue of safe ownership model as 3rd party applications would not be able to accidentally wipe Cilium programs, even if they are not BPF link aware. Earlier attempts [4] have tried to integrate BPF links into core tc machinery to solve cls_bpf, which has been intrusive to the generic tc kernel API with extensions only specific to cls_bpf and suboptimal/complex since cls_bpf could be wiped from the qdisc also. Locking a tc BPF program in place this way, is getting into layering hacks given the two object models are vastly different. We instead implemented the tcx (tc 'express') layer which is an fd-based tc BPF attach API, so that the BPF link implementation blends in naturally similar to other link types which are fd-based and without the need for changing core tc internal APIs. BPF programs for tc can then be successively migrated from classic cls_bpf to the new tc BPF link without needing to change the program's source code, just the BPF loader mechanics for attaching is sufficient. For the current tc framework, there is no change in behavior with this change and neither does this change touch on tc core kernel APIs. The gist of this patch is that the ingress and egress hook have a lightweight, qdisc-less extension for BPF to attach its tc BPF programs, in other words, a minimal entry point for tc BPF. The name tcx has been suggested from discussion of earlier revisions of this work as a good fit, and to more easily differ between the classic cls_bpf attachment and the fd-based one. For the ingress and egress tcx points, the device holds a cache-friendly array with program pointers which is separated from control plane (slow-path) data. Earlier versions of this work used priority to determine ordering and expression of dependencies similar as with classic tc, but it was challenged that for something more future-proof a better user experience is required. Hence this resulted in the design and development of the generic attach/detach/query API for multi-progs. See prior patch with its discussion on the API design. tcx is the first user and later we plan to integrate also others, for example, one candidate is multi-prog support for XDP which would benefit and have the same 'look and feel' from API perspective. The goal with tcx is to have maximum compatibility to existing tc BPF programs, so they don't need to be rewritten specifically. Compatibility to call into classic tcf_classify() is also provided in order to allow successive migration or both to cleanly co-exist where needed given its all one logical tc layer and the tcx plus classic tc cls/act build one logical overall processing pipeline. tcx supports the simplified return codes TCX_NEXT which is non-terminating (go to next program) and terminating ones with TCX_PASS, TCX_DROP, TCX_REDIRECT. The fd-based API is behind a static key, so that when unused the code is also not entered. The struct tcx_entry's program array is currently static, but could be made dynamic if necessary at a point in future. The a/b pair swap design has been chosen so that for detachment there are no allocations which otherwise could fail. The work has been tested with tc-testing selftest suite which all passes, as well as the tc BPF tests from the BPF CI, and also with Cilium's L4LB. Thanks also to Nikolay Aleksandrov and Martin Lau for in-depth early reviews of this work. [0] https://lpc.events/event/16/contributions/1353/ [1] https://lore.kernel.org/bpf/CAEf4BzbokCJN33Nw_kg82sO=xppXnKWEncGTWCTB9vGCmLB6pw@mail.gmail.com [2] https://colocatedeventseu2023.sched.com/event/1Jo6O/tales-from-an-ebpf-programs-murder-mystery-hemanth-malla-guillaume-fournier-datadog [3] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf [4] https://lore.kernel.org/bpf/20210604063116.234316-1-memxor@gmail.com Signed-off-by: Daniel Borkmann Acked-by: Jakub Kicinski Link: https://lore.kernel.org/r/20230719140858.13224-3-daniel@iogearbox.net Signed-off-by: Alexei Starovoitov MAINTAINERS | 4 +- include/linux/bpf_mprog.h | 9 ++ include/linux/netdevice.h | 15 +- include/linux/skbuff.h | 4 +- include/net/sch_generic.h | 2 +- include/net/tcx.h | 206 ++++++++++++++++++++++++ include/uapi/linux/bpf.h | 34 +++- kernel/bpf/Kconfig | 1 + kernel/bpf/Makefile | 1 + kernel/bpf/syscall.c | 82 ++++++++-- kernel/bpf/tcx.c | 348 +++++++++++++++++++++++++++++++++++++++++ net/Kconfig | 5 + net/core/dev.c | 265 +++++++++++++++++++------------ net/core/filter.c | 4 +- net/sched/Kconfig | 4 +- net/sched/sch_ingress.c | 61 +++++++- tools/include/uapi/linux/bpf.h | 34 +++- 17 files changed, 935 insertions(+), 144 deletions(-) create mode 100644 include/net/tcx.h create mode 100644 kernel/bpf/tcx.c accumulated error probability: 0.00 culprit signature: 00c3e06d4ef86913dcba41390a7cf91659c7332620641bc23932cdecb55aaccd parent signature: 53788cea7907e38be7bdb8267bb985531804710e04ae29aed9571ce4b72802c7 revisions tested: 22, total time: 6h44m7.317676864s (build: 3h2m3.359851263s, test: 3h26m40.197536236s) first bad commit: e420bed025071a623d2720a92bc2245c84757ecb bpf: Add fd-based tcx multi-prog infra with link support recipients (to): ["ast@kernel.org" "daniel@iogearbox.net" "kuba@kernel.org"] recipients (cc): [] crash: WARNING: ODEBUG bug in ingress_destroy ------------[ cut here ]------------ ODEBUG: activate active (active state 1) object: ffff888102fdc800 object type: rcu_head hint: 0x0 WARNING: CPU: 1 PID: 3383 at lib/debugobjects.c:514 debug_print_object+0x7a/0xa0 lib/debugobjects.c:514 Modules linked in: CPU: 1 PID: 3383 Comm: syz-executor.0 Not tainted 6.5.0-rc1-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/12/2023 RIP: 0010:debug_print_object+0x7a/0xa0 lib/debugobjects.c:514 Code: 01 4c 8b 4d 00 48 c7 c7 90 57 3b 83 89 15 be 2f 5b 03 8b 53 10 8b 4b 14 50 4c 8b 43 18 48 8b 14 d5 00 aa e9 82 e8 26 9a 75 ff <0f> 0b 58 83 05 40 17 ff 01 01 48 83 c4 08 5b 5d c3 83 05 32 17 ff RSP: 0018:ffffc90001adb768 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff88810271e410 RCX: 0000000000000000 RDX: ffff888103293680 RSI: ffffffff8116cf61 RDI: 0000000000000001 RBP: ffffffff82e2a6a0 R08: 0000000000000001 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000001 R12: ffff888102fdc800 R13: ffffffff850ce5f0 R14: ffffffff850ce5e8 R15: ffffffff82e2a6a0 FS: 00007fcd1edad6c0(0000) GS:ffff888237d00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000108ba4000 CR4: 00000000003506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: debug_object_activate+0x1fd/0x280 lib/debugobjects.c:733 debug_rcu_head_queue kernel/rcu/rcu.h:226 [inline] kvfree_call_rcu+0x30/0x3b0 kernel/rcu/tree.c:3359 tcx_entry_free include/net/tcx.h:96 [inline] ingress_destroy+0x152/0x250 net/sched/sch_ingress.c:127 __qdisc_destroy+0x48/0x210 net/sched/sch_generic.c:1063 qdisc_destroy+0x2e/0x40 net/sched/sch_generic.c:1078 qdisc_graft+0x386/0xad0 net/sched/sch_api.c:1132 tc_modify_qdisc+0x6f9/0xcf0 net/sched/sch_api.c:1731 rtnetlink_rcv_msg+0x287/0x670 net/core/rtnetlink.c:6424 netlink_rcv_skb+0x91/0x1d0 net/netlink/af_netlink.c:2546 netlink_unicast_kernel net/netlink/af_netlink.c:1339 [inline] netlink_unicast+0x2c0/0x410 net/netlink/af_netlink.c:1365 netlink_sendmsg+0x3a5/0x740 net/netlink/af_netlink.c:1911 sock_sendmsg_nosec net/socket.c:725 [inline] sock_sendmsg+0x56/0xb0 net/socket.c:748 ____sys_sendmsg+0x368/0x470 net/socket.c:2494 ___sys_sendmsg+0xc9/0x130 net/socket.c:2548 __sys_sendmsg+0xa6/0x120 net/socket.c:2577 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x38/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd RIP: 0033:0x7fcd1f22ab29 Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007fcd1edad0c8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e RAX: ffffffffffffffda RBX: 00007fcd1f349f80 RCX: 00007fcd1f22ab29 RDX: 0000000000000000 RSI: 0000000020000080 RDI: 0000000000000003 RBP: 00007fcd1f27647a R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000006 R14: 00007fcd1f349f80 R15: 00007ffed6a51528