bisecting cause commit starting from 4c375272fb0ba19288e323928d1f6937e368851f building syzkaller on 0230ba3e7ee638765ace8e2c3b436e703017b46c testing commit 4c375272fb0ba19288e323928d1f6937e368851f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: f6ec078c6177900a0a386d7eae60392554ce6d8f0f19d6b898289c3a76e8b12d all runs: crashed: WARNING: suspicious RCU usage in __dev_queue_xmit testing release v5.15 testing commit 8bb7eca972ad531c9b149c0a51ab43a417385813 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 23b867ff5df3d6cb6b498368a18ae79abf8004bbd96dea98ad38033811273ce5 all runs: OK # git bisect start 4c375272fb0ba19288e323928d1f6937e368851f 8bb7eca972ad531c9b149c0a51ab43a417385813 Bisecting: 7422 revisions left to test after this (roughly 13 steps) [a602285ac11b019e9ce7c3907328e9f95f4967f0] Merge branch 'per_signal_struct_coredumps-for-v5.16' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace testing commit a602285ac11b019e9ce7c3907328e9f95f4967f0 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: b1854b2631b224b50e1ec8faf50edfe55fa9ec4ee326aa6a06e15e42795169df all runs: crashed: WARNING: suspicious RCU usage in __dev_queue_xmit # git bisect bad a602285ac11b019e9ce7c3907328e9f95f4967f0 Bisecting: 2833 revisions left to test after this (roughly 12 steps) [fc02cb2b37fe2cbf1d3334b9f0f0eab9431766c4] Merge tag 'net-next-for-5.16' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next testing commit fc02cb2b37fe2cbf1d3334b9f0f0eab9431766c4 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5aca587dcf3012cb326617f004c37f0c85a595286744cb9e486117aa2f8ab8c2 all runs: crashed: WARNING: suspicious RCU usage in __dev_queue_xmit # git bisect bad fc02cb2b37fe2cbf1d3334b9f0f0eab9431766c4 Bisecting: 2251 revisions left to test after this (roughly 11 steps) [b7b98f868987cd3e86c9bd9a6db048614933d7a0] Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next testing commit b7b98f868987cd3e86c9bd9a6db048614933d7a0 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a98343e23dd1dcd0335ca7d914e36bd0aa6fdb022225b6ae90e8d5d31f60b412 all runs: crashed: WARNING: suspicious RCU usage in __dev_queue_xmit # git bisect bad b7b98f868987cd3e86c9bd9a6db048614933d7a0 Bisecting: 1156 revisions left to test after this (roughly 10 steps) [9fd3d5dced976640f588e0a866b9611db2d2cb37] net: ethernet: ave: Add compatible string and SoC-dependent data for NX1 SoC testing commit 9fd3d5dced976640f588e0a866b9611db2d2cb37 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 07e96f8e18ee39846ed71e239ab80306dfcd3dc8584d99e1423a26cbdc788e71 all runs: OK # git bisect good 9fd3d5dced976640f588e0a866b9611db2d2cb37 Bisecting: 578 revisions left to test after this (roughly 9 steps) [1c375ffb2efab992b74fb1801c2e0bb2051a6e6e] mlxsw: spectrum_router: Expose RIF MAC profiles to devlink resource testing commit 1c375ffb2efab992b74fb1801c2e0bb2051a6e6e compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a568a1ea9a5f3f28008c52372344d1b50d52c3f9e8f1b339f28d3a684888e8da all runs: crashed: WARNING in nsim_dev_reload_destroy # git bisect bad 1c375ffb2efab992b74fb1801c2e0bb2051a6e6e Bisecting: 362 revisions left to test after this (roughly 8 steps) [9bc0b1aa8b7e54d62082749fc5404660690d17ce] Merge tag 'mt76-for-kvalo-2021-10-20' of https://github.com/nbd168/wireless testing commit 9bc0b1aa8b7e54d62082749fc5404660690d17ce compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2798c32f12738518c7ded89f1e74e01aa0aeadd1a108547ad9ebbfb9ce5e2410 all runs: OK # git bisect good 9bc0b1aa8b7e54d62082749fc5404660690d17ce Bisecting: 194 revisions left to test after this (roughly 8 steps) [07591ebec3cf2d6b78cb9b51a5a6f3ca731ec375] Merge branch 'net-don-t-write-directly-to-netdev-dev_addr' testing commit 07591ebec3cf2d6b78cb9b51a5a6f3ca731ec375 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 191d007342c814820fc112104a3ebab26a6bf40566b275279bfbe09ed16630b0 all runs: crashed: WARNING: suspicious RCU usage in __dev_queue_xmit # git bisect bad 07591ebec3cf2d6b78cb9b51a5a6f3ca731ec375 Bisecting: 83 revisions left to test after this (roughly 6 steps) [e22db7bd552f7f7f19fe4ef60abfb7e7b364e3a8] net: sched: Allow statistics reads from softirq. testing commit e22db7bd552f7f7f19fe4ef60abfb7e7b364e3a8 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 32f0933a1b636033fedc50795ee94d1980abc860fc5f63126bb46771fd8f126c all runs: crashed: WARNING: suspicious RCU usage in __dev_queue_xmit # git bisect bad e22db7bd552f7f7f19fe4ef60abfb7e7b364e3a8 Bisecting: 41 revisions left to test after this (roughly 5 steps) [8e25a2bc6687cd45aef909d6d862718745b5a3fc] net/mlx5: Lag, add support to create TTC tables for LAG port selection testing commit 8e25a2bc6687cd45aef909d6d862718745b5a3fc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6cc8114dd440cb0768ed7ebfcaa13d66b9fc45680d5b9fa69a49f32f2195cd86 all runs: crashed: WARNING: suspicious RCU usage in __dev_queue_xmit # git bisect bad 8e25a2bc6687cd45aef909d6d862718745b5a3fc Bisecting: 26 revisions left to test after this (roughly 4 steps) [ffdd33dd9c12a8c263f78d778066709ef94671f9] netfilter: core: Fix clang warnings about unused static inlines testing commit ffdd33dd9c12a8c263f78d778066709ef94671f9 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 69dee6ae1df41440812f9e92ad892ae16bde98271a3a43657534be8147f2b58d all runs: crashed: WARNING: suspicious RCU usage in __dev_queue_xmit # git bisect bad ffdd33dd9c12a8c263f78d778066709ef94671f9 Bisecting: 7 revisions left to test after this (roughly 3 steps) [8844e01062ddd8196c4550df9803cc1835d123c2] netfilter: iptables: allow use of ipt_do_table as hookfn testing commit 8844e01062ddd8196c4550df9803cc1835d123c2 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 3200b9fc1d28750b27231652d47b29aa4b747f6fbcabe49a05bf432500145a19 all runs: crashed: WARNING: suspicious RCU usage in __dev_queue_xmit # git bisect bad 8844e01062ddd8196c4550df9803cc1835d123c2 Bisecting: 3 revisions left to test after this (roughly 2 steps) [7463acfbe52ae8b7e0ea6890c1886b3f8ba8bddd] netfilter: Rename ingress hook include file testing commit 7463acfbe52ae8b7e0ea6890c1886b3f8ba8bddd compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6458de55c9acdf4ba1b3f3e982703203148610969a1d17a7e2aa3c87c48b20cb all runs: OK # git bisect good 7463acfbe52ae8b7e0ea6890c1886b3f8ba8bddd Bisecting: 1 revision left to test after this (roughly 1 step) [42df6e1d221dddc0f2acf2be37e68d553ad65f96] netfilter: Introduce egress hook testing commit 42df6e1d221dddc0f2acf2be37e68d553ad65f96 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d3a6bcce6adab0432df23a9395231deda4911008da948cd76b0dd2662c10665f all runs: crashed: WARNING: suspicious RCU usage in __dev_queue_xmit # git bisect bad 42df6e1d221dddc0f2acf2be37e68d553ad65f96 Bisecting: 0 revisions left to test after this (roughly 0 steps) [17d20784223d52bf1671f984c9e8d5d9b8ea171b] netfilter: Generalize ingress hook include file testing commit 17d20784223d52bf1671f984c9e8d5d9b8ea171b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6458de55c9acdf4ba1b3f3e982703203148610969a1d17a7e2aa3c87c48b20cb all runs: OK # git bisect good 17d20784223d52bf1671f984c9e8d5d9b8ea171b 42df6e1d221dddc0f2acf2be37e68d553ad65f96 is the first bad commit commit 42df6e1d221dddc0f2acf2be37e68d553ad65f96 Author: Lukas Wunner Date: Fri Oct 8 22:06:03 2021 +0200 netfilter: Introduce egress hook Support classifying packets with netfilter on egress to satisfy user requirements such as: * outbound security policies for containers (Laura) * filtering and mangling intra-node Direct Server Return (DSR) traffic on a load balancer (Laura) * filtering locally generated traffic coming in through AF_PACKET, such as local ARP traffic generated for clustering purposes or DHCP (Laura; the AF_PACKET plumbing is contained in a follow-up commit) * L2 filtering from ingress and egress for AVB (Audio Video Bridging) and gPTP with nftables (Pablo) * in the future: in-kernel NAT64/NAT46 (Pablo) The egress hook introduced herein complements the ingress hook added by commit e687ad60af09 ("netfilter: add netfilter ingress hook after handle_ing() under unique static key"). A patch for nftables to hook up egress rules from user space has been submitted separately, so users may immediately take advantage of the feature. Alternatively or in addition to netfilter, packets can be classified with traffic control (tc). On ingress, packets are classified first by tc, then by netfilter. On egress, the order is reversed for symmetry. Conceptually, tc and netfilter can be thought of as layers, with netfilter layered above tc. Traffic control is capable of redirecting packets to another interface (man 8 tc-mirred). E.g., an ingress packet may be redirected from the host namespace to a container via a veth connection: tc ingress (host) -> tc egress (veth host) -> tc ingress (veth container) In this case, netfilter egress classifying is not performed when leaving the host namespace! That's because the packet is still on the tc layer. If tc redirects the packet to a physical interface in the host namespace such that it leaves the system, the packet is never subjected to netfilter egress classifying. That is only logical since it hasn't passed through netfilter ingress classifying either. Packets can alternatively be redirected at the netfilter layer using nft fwd. Such a packet *is* subjected to netfilter egress classifying since it has reached the netfilter layer. Internally, the skb->nf_skip_egress flag controls whether netfilter is invoked on egress by __dev_queue_xmit(). Because __dev_queue_xmit() may be called recursively by tunnel drivers such as vxlan, the flag is reverted to false after sch_handle_egress(). This ensures that netfilter is applied both on the overlay and underlying network. Interaction between tc and netfilter is possible by setting and querying skb->mark. If netfilter egress classifying is not enabled on any interface, it is patched out of the data path by way of a static_key and doesn't make a performance difference that is discernible from noise: Before: 1537 1538 1538 1537 1538 1537 Mb/sec After: 1536 1534 1539 1539 1539 1540 Mb/sec Before + tc accept: 1418 1418 1418 1419 1419 1418 Mb/sec After + tc accept: 1419 1424 1418 1419 1422 1420 Mb/sec Before + tc drop: 1620 1619 1619 1619 1620 1620 Mb/sec After + tc drop: 1616 1624 1625 1624 1622 1619 Mb/sec When netfilter egress classifying is enabled on at least one interface, a minimal performance penalty is incurred for every egress packet, even if the interface it's transmitted over doesn't have any netfilter egress rules configured. That is caused by checking dev->nf_hooks_egress against NULL. Measurements were performed on a Core i7-3615QM. Commands to reproduce: ip link add dev foo type dummy ip link set dev foo up modprobe pktgen echo "add_device foo" > /proc/net/pktgen/kpktgend_3 samples/pktgen/pktgen_bench_xmit_mode_queue_xmit.sh -i foo -n 400000000 -m "11:11:11:11:11:11" -d 1.1.1.1 Accept all traffic with tc: tc qdisc add dev foo clsact tc filter add dev foo egress bpf da bytecode '1,6 0 0 0,' Drop all traffic with tc: tc qdisc add dev foo clsact tc filter add dev foo egress bpf da bytecode '1,6 0 0 2,' Apply this patch when measuring packet drops to avoid errors in dmesg: https://lore.kernel.org/netdev/a73dda33-57f4-95d8-ea51-ed483abd6a7a@iogearbox.net/ Signed-off-by: Lukas Wunner Cc: Laura García Liébana Cc: John Fastabend Cc: Daniel Borkmann Cc: Alexei Starovoitov Cc: Eric Dumazet Cc: Thomas Graf Signed-off-by: Pablo Neira Ayuso drivers/net/ifb.c | 3 ++ include/linux/netdevice.h | 4 ++ include/linux/netfilter_netdev.h | 86 ++++++++++++++++++++++++++++++++++++++++ include/linux/skbuff.h | 4 ++ include/uapi/linux/netfilter.h | 1 + net/core/dev.c | 15 ++++++- net/netfilter/Kconfig | 11 +++++ net/netfilter/core.c | 34 ++++++++++++++-- net/netfilter/nfnetlink_hook.c | 16 ++++++-- net/netfilter/nft_chain_filter.c | 4 +- 10 files changed, 168 insertions(+), 10 deletions(-) culprit signature: d3a6bcce6adab0432df23a9395231deda4911008da948cd76b0dd2662c10665f parent signature: 6458de55c9acdf4ba1b3f3e982703203148610969a1d17a7e2aa3c87c48b20cb revisions tested: 16, total time: 3h24m24.461635844s (build: 1h46m12.743545101s, test: 1h36m23.908551511s) first bad commit: 42df6e1d221dddc0f2acf2be37e68d553ad65f96 netfilter: Introduce egress hook recipients (to): ["coreteam@netfilter.org" "davem@davemloft.net" "fw@strlen.de" "kadlec@netfilter.org" "kuba@kernel.org" "lukas@wunner.de" "netdev@vger.kernel.org" "netfilter-devel@vger.kernel.org" "pablo@netfilter.org" "pablo@netfilter.org"] recipients (cc): ["alobakin@pm.me" "andrii@kernel.org" "ast@kernel.org" "atenart@kernel.org" "bpf@vger.kernel.org" "daniel@iogearbox.net" "john.fastabend@gmail.com" "jonathan.lemon@gmail.com" "kafai@fb.com" "kpsingh@kernel.org" "linux-kernel@vger.kernel.org" "songliubraving@fb.com" "yhs@fb.com"] crash: WARNING: suspicious RCU usage in __dev_queue_xmit wlan1: Created IBSS using preconfigured BSSID 50:50:50:50:50:50 wlan1: Creating new IBSS network, BSSID 50:50:50:50:50:50 ============================= WARNING: suspicious RCU usage 5.15.0-rc3-syzkaller #0 Not tainted ----------------------------- include/linux/netfilter_netdev.h:97 suspicious rcu_dereference_check() usage! other info that might help us debug this: rcu_scheduler_active = 2, debug_locks = 1 3 locks held by kworker/u4:6/2574: #0: ffff88814794a138 ((wq_completion)bat_events){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: ffff88814794a138 ((wq_completion)bat_events){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline] #0: ffff88814794a138 ((wq_completion)bat_events){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1198 [inline] #0: ffff88814794a138 ((wq_completion)bat_events){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:634 [inline] #0: ffff88814794a138 ((wq_completion)bat_events){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:661 [inline] #0: ffff88814794a138 ((wq_completion)bat_events){+.+.}-{0:0}, at: process_one_work+0x7a4/0x1450 kernel/workqueue.c:2268 #1: ffffc9000b14fdb8 ((work_completion)(&(&forw_packet_aggr->delayed_work)->work)){+.+.}-{0:0}, at: process_one_work+0x7d1/0x1450 kernel/workqueue.c:2272 #2: ffffffff8ab78860 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1cc/0x3140 net/core/dev.c:4140 stack backtrace: CPU: 1 PID: 2574 Comm: kworker/u4:6 Not tainted 5.15.0-rc3-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: bat_events batadv_iv_send_outstanding_bat_ogm_packet Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x57/0x7d lib/dump_stack.c:106 nf_hook_egress include/linux/netfilter_netdev.h:97 [inline] __dev_queue_xmit+0x2756/0x3140 net/core/dev.c:4157 batadv_send_skb_packet+0x46b/0x590 net/batman-adv/send.c:108 batadv_iv_ogm_send_to_if net/batman-adv/bat_iv_ogm.c:393 [inline] batadv_iv_ogm_emit net/batman-adv/bat_iv_ogm.c:421 [inline] batadv_iv_send_outstanding_bat_ogm_packet+0x54c/0x8f0 net/batman-adv/bat_iv_ogm.c:1701 process_one_work+0x87f/0x1450 kernel/workqueue.c:2297 worker_thread+0x598/0x1040 kernel/workqueue.c:2444 kthread+0x38b/0x460 kernel/kthread.c:319 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295