bisecting cause commit starting from 38b5133ad607ecdcc8d24906d1ac9cc8df41acd5 building syzkaller on f689d40a08ae1c9d73d043e7a2e807bc4022931b testing commit 38b5133ad607ecdcc8d24906d1ac9cc8df41acd5 with gcc (GCC) 10.2.1 20210217 kernel signature: 3f9135f86e99b43d9c8aa2db25827b670542bd9f7a4eb8b3a7501eb54af47590 all runs: crashed: BUG: unable to handle kernel NULL pointer dereference in htb_select_queue testing release v5.10 testing commit 2c85ebc57b3e1817b6ce1a6b703928e113a90442 with gcc (GCC) 10.2.1 20210217 kernel signature: ad0020076c568fd04e3712999367961eadbdf96cf6f928f88e3fec927cee834c all runs: OK # git bisect start 38b5133ad607ecdcc8d24906d1ac9cc8df41acd5 2c85ebc57b3e1817b6ce1a6b703928e113a90442 Bisecting: 8695 revisions left to test after this (roughly 13 steps) [005b2a9dc819a1265a8c765595f8f6d88d6173d9] Merge tag 'tif-task_work.arch-2020-12-14' of git://git.kernel.dk/linux-block testing commit 005b2a9dc819a1265a8c765595f8f6d88d6173d9 with gcc (GCC) 10.2.1 20210217 kernel signature: dcde3cf0af992b52fa9a5a77cae45152618de4bea64cb249e6da4fbd339396c7 all runs: OK # git bisect good 005b2a9dc819a1265a8c765595f8f6d88d6173d9 Bisecting: 4352 revisions left to test after this (roughly 12 steps) [14571d5f22d3f7f6ecb97e037a2e346b3fb488bd] Merge tag 'devicetree-fixes-for-5.11-1' of git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux testing commit 14571d5f22d3f7f6ecb97e037a2e346b3fb488bd with gcc (GCC) 10.2.1 20210217 kernel signature: a57fb85f80cc6b909786857b6c24462c53807c47625c8b271aebf54338285c60 all runs: OK # git bisect good 14571d5f22d3f7f6ecb97e037a2e346b3fb488bd Bisecting: 2173 revisions left to test after this (roughly 11 steps) [6157ce59bf318bd4ee23769c613cf5628d7f457b] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm testing commit 6157ce59bf318bd4ee23769c613cf5628d7f457b with gcc (GCC) 10.2.1 20210217 kernel signature: 4f038e184e3687fc4978880be78b0b1dd442162435d7672d7ebcb927c05b8e58 all runs: OK # git bisect good 6157ce59bf318bd4ee23769c613cf5628d7f457b Bisecting: 1086 revisions left to test after this (roughly 10 steps) [adbb4fb028452b1b0488a1a7b66ab856cdf20715] Merge branch 'implement-kthread-based-napi-poll' testing commit adbb4fb028452b1b0488a1a7b66ab856cdf20715 with gcc (GCC) 10.2.1 20210217 kernel signature: 5dc701ff3b90776ce0b4d4561ff417101443b33c1f65dad32c6600fa38254fb5 all runs: crashed: BUG: unable to handle kernel NULL pointer dereference in htb_select_queue # git bisect bad adbb4fb028452b1b0488a1a7b66ab856cdf20715 Bisecting: 543 revisions left to test after this (roughly 9 steps) [6d70cd2a42095db3e123271672cc996d12d0a194] Merge branch 'bnxt_en-error-recovery-improvements' testing commit 6d70cd2a42095db3e123271672cc996d12d0a194 with gcc (GCC) 10.2.1 20210217 kernel signature: 491f90604d0b9dbf255133b862267c94936245e0e781faebc9a8f41bffda49a0 all runs: crashed: BUG: unable to handle kernel NULL pointer dereference in htb_select_queue # git bisect bad 6d70cd2a42095db3e123271672cc996d12d0a194 Bisecting: 271 revisions left to test after this (roughly 8 steps) [74401946bdad6eb812d2d3c77f1ace849f963a6a] net: ipa: use usleep_range() 65;6003;1c The use of msleep() for small periods (less than 20 milliseconds) is not recommended because the actual delay can be much different than expected. testing commit 74401946bdad6eb812d2d3c77f1ace849f963a6a with gcc (GCC) 10.2.1 20210217 kernel signature: 7d3f2738fc8979ece8e50129f71d49cab1255b5b10d836f1569e6470d8d1f337 all runs: OK # git bisect good 74401946bdad6eb812d2d3c77f1ace849f963a6a Bisecting: 135 revisions left to test after this (roughly 7 steps) [b3228c74e0d24976604e8550e9cccb83135f5302] dt-binding: ti: am65x-cpts: add assigned-clock and power-domains props testing commit b3228c74e0d24976604e8550e9cccb83135f5302 with gcc (GCC) 10.2.1 20210217 kernel signature: 656c478e07d4883dbbef663fb6889acf9a662d11732f34e06ff0c6881f4ca797 all runs: OK # git bisect good b3228c74e0d24976604e8550e9cccb83135f5302 Bisecting: 67 revisions left to test after this (roughly 6 steps) [40dc9416cc957ac8b74d09550a808fabfd4435f8] mptcp: schedule work for better snd subflow selection testing commit 40dc9416cc957ac8b74d09550a808fabfd4435f8 with gcc (GCC) 10.2.1 20210217 kernel signature: 3b887fcecdaa9416b6975c2f20d078d29ba001c7093bfd47fb9f1b5c2381d224 all runs: OK # git bisect good 40dc9416cc957ac8b74d09550a808fabfd4435f8 Bisecting: 33 revisions left to test after this (roughly 5 steps) [5725593e6f182607993364f56ab1c0468d68016f] net: ipa: repurpose gsi_irq_ieob_disable() testing commit 5725593e6f182607993364f56ab1c0468d68016f with gcc (GCC) 10.2.1 20210217 kernel signature: fee27fec596e0a2d1d7641ade7d99769d2c7a5cbf579985e802a4276dd974fdb all runs: crashed: BUG: unable to handle kernel NULL pointer dereference in htb_select_queue # git bisect bad 5725593e6f182607993364f56ab1c0468d68016f Bisecting: 16 revisions left to test after this (roughly 4 steps) [d5a1022283c3b0baa252506b34178266a4c0db4d] net: bridge: multicast: mark IGMPv3/MLDv2 fast-leave deletes testing commit d5a1022283c3b0baa252506b34178266a4c0db4d with gcc (GCC) 10.2.1 20210217 kernel signature: 8c2d23293463ed40488dff38bc9f8b79a3b6ede85750dfd7f3f0b4487d6a1d94 all runs: OK # git bisect good d5a1022283c3b0baa252506b34178266a4c0db4d Bisecting: 8 revisions left to test after this (roughly 3 steps) [83271586249c8ecf8458834864c827f67ad57773] sch_htb: Stats for offloaded HTB testing commit 83271586249c8ecf8458834864c827f67ad57773 with gcc (GCC) 10.2.1 20210217 kernel signature: 20b30325fc600e148c89077eb9a235cadfaf5a6525f885d42dbcd26d692b6181 all runs: crashed: BUG: unable to handle kernel NULL pointer dereference in htb_select_queue # git bisect bad 83271586249c8ecf8458834864c827f67ad57773 Bisecting: 3 revisions left to test after this (roughly 2 steps) [04a886372a2052078fa72b4ecb23f0ac8094f654] Merge branch 'tcp-add-cmsg-rx-timestamps-to-rx-zerocopy' testing commit 04a886372a2052078fa72b4ecb23f0ac8094f654 with gcc (GCC) 10.2.1 20210217 kernel signature: 6cc0c9b24961e14550cf872846e2250f7bad0c5c95862a6776aaba24e3cc825f all runs: OK # git bisect good 04a886372a2052078fa72b4ecb23f0ac8094f654 Bisecting: 1 revision left to test after this (roughly 1 step) [4dd78a73738afa92d33a226ec477b42938b31c83] net: sched: Add extack to Qdisc_class_ops.delete testing commit 4dd78a73738afa92d33a226ec477b42938b31c83 with gcc (GCC) 10.2.1 20210217 kernel signature: 34cdad9162923a7640327100aea41831f6da1c265f807b1e89d0a295a2d85cb9 all runs: OK # git bisect good 4dd78a73738afa92d33a226ec477b42938b31c83 Bisecting: 0 revisions left to test after this (roughly 0 steps) [d03b195b5aa015f6c11988b86a3625f8d5dbac52] sch_htb: Hierarchical QoS hardware offload testing commit d03b195b5aa015f6c11988b86a3625f8d5dbac52 with gcc (GCC) 10.2.1 20210217 kernel signature: d5aa260aab51a4c8b621689b0310c14ee36f870a0414f75d689b3d0118ddb21f all runs: crashed: BUG: unable to handle kernel NULL pointer dereference in htb_select_queue # git bisect bad d03b195b5aa015f6c11988b86a3625f8d5dbac52 d03b195b5aa015f6c11988b86a3625f8d5dbac52 is the first bad commit commit d03b195b5aa015f6c11988b86a3625f8d5dbac52 Author: Maxim Mikityanskiy Date: Tue Jan 19 14:08:13 2021 +0200 sch_htb: Hierarchical QoS hardware offload HTB doesn't scale well because of contention on a single lock, and it also consumes CPU. This patch adds support for offloading HTB to hardware that supports hierarchical rate limiting. In the offload mode, HTB passes control commands to the driver using ndo_setup_tc. The driver has to replicate the whole hierarchy of classes and their settings (rate, ceil) in the NIC. Every modification of the HTB tree caused by the admin results in ndo_setup_tc being called. After this setup, the HTB algorithm is done completely in the NIC. An SQ (send queue) is created for every leaf class and attached to the hierarchy, so that the NIC can calculate and obey aggregated rate limits, too. In the future, it can be changed, so that multiple SQs will back a single leaf class. ndo_select_queue is responsible for selecting the right queue that serves the traffic class of each packet. The data path works as follows: a packet is classified by clsact, the driver selects a hardware queue according to its class, and the packet is enqueued into this queue's qdisc. This solution addresses two main problems of scaling HTB: 1. Contention by flow classification. Currently the filters are attached to the HTB instance as follows: # tc filter add dev eth0 parent 1:0 protocol ip flower dst_port 80 classid 1:10 It's possible to move classification to clsact egress hook, which is thread-safe and lock-free: # tc filter add dev eth0 egress protocol ip flower dst_port 80 action skbedit priority 1:10 This way classification still happens in software, but the lock contention is eliminated, and it happens before selecting the TX queue, allowing the driver to translate the class to the corresponding hardware queue in ndo_select_queue. Note that this is already compatible with non-offloaded HTB and doesn't require changes to the kernel nor iproute2. 2. Contention by handling packets. HTB is not multi-queue, it attaches to a whole net device, and handling of all packets takes the same lock. When HTB is offloaded, it registers itself as a multi-queue qdisc, similarly to mq: HTB is attached to the netdev, and each queue has its own qdisc. Some features of HTB may be not supported by some particular hardware, for example, the maximum number of classes may be limited, the granularity of rate and ceil parameters may be different, etc. - so, the offload is not enabled by default, a new parameter is used to enable it: # tc qdisc replace dev eth0 root handle 1: htb offload Signed-off-by: Maxim Mikityanskiy Reviewed-by: Tariq Toukan Signed-off-by: Jakub Kicinski include/linux/netdevice.h | 1 + include/net/pkt_cls.h | 36 +++ include/uapi/linux/pkt_sched.h | 1 + net/sched/sch_htb.c | 501 +++++++++++++++++++++++++++++++++-- tools/include/uapi/linux/pkt_sched.h | 1 + 5 files changed, 512 insertions(+), 28 deletions(-) culprit signature: d5aa260aab51a4c8b621689b0310c14ee36f870a0414f75d689b3d0118ddb21f parent signature: 34cdad9162923a7640327100aea41831f6da1c265f807b1e89d0a295a2d85cb9 revisions tested: 16, total time: 3h34m37.439598781s (build: 1h45m15.325309619s, test: 1h47m51.378092644s) first bad commit: d03b195b5aa015f6c11988b86a3625f8d5dbac52 sch_htb: Hierarchical QoS hardware offload recipients (to): ["kuba@kernel.org" "maximmi@mellanox.com" "tariqt@nvidia.com"] recipients (cc): [] crash: BUG: unable to handle kernel NULL pointer dereference in htb_select_queue BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor instruction fetch in kernel mode #PF: error_code(0x0010) - not-present page PGD 12a5a067 P4D 12a5a067 PUD 2a5a0067 PMD 0 Oops: 0010 [#1] PREEMPT SMP KASAN CPU: 1 PID: 12857 Comm: syz-executor.5 Not tainted 5.11.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:0x0 Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6. RSP: 0018:ffffc90002e47508 EFLAGS: 00010246 RAX: dffffc0000000000 RBX: 1ffff920005c8ea3 RCX: 0000000000000000 RDX: ffffc90002e47548 RSI: 0000000000000012 RDI: ffff888012b52000 RBP: ffff888012b52000 R08: ffff888019c30c18 R09: 0000000000000000 R10: ffffffff8979d8e8 R11: fffff520005c8ea1 R12: ffffffff890469c0 R13: 00000000ffff0000 R14: ffff888019c30c1c R15: ffff888019e7a000 FS: 00007f5bf96f2700(0000) GS:ffff8880b9f00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffffffffd6 CR3: 00000000143db000 CR4: 00000000001506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: htb_offload net/sched/sch_htb.c:1010 [inline] htb_select_queue+0x174/0x290 net/sched/sch_htb.c:1303 tc_modify_qdisc+0x35d/0x1630 net/sched/sch_api.c:1657 rtnetlink_rcv_msg+0x32f/0x860 net/core/rtnetlink.c:5553 netlink_rcv_skb+0x118/0x370 net/netlink/af_netlink.c:2494 netlink_unicast_kernel net/netlink/af_netlink.c:1304 [inline] netlink_unicast+0x42e/0x700 net/netlink/af_netlink.c:1330 netlink_sendmsg+0x70e/0xbe0 net/netlink/af_netlink.c:1919 sock_sendmsg_nosec net/socket.c:652 [inline] sock_sendmsg+0xab/0xe0 net/socket.c:672 ____sys_sendmsg+0x5bf/0x7a0 net/socket.c:2345 ___sys_sendmsg+0xd3/0x150 net/socket.c:2399 __sys_sendmsg+0xb2/0x140 net/socket.c:2432 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x466019 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f5bf96f2188 EFLAGS: 00000246 ORIG_RAX: 000000000000002e RAX: ffffffffffffffda RBX: 000000000056bf60 RCX: 0000000000466019 RDX: 0000000000000000 RSI: 00000000200007c0 RDI: 0000000000000004 RBP: 00000000004bd067 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf60 R13: 00007ffccab5a9ef R14: 00007f5bf96f2300 R15: 0000000000022000 Modules linked in: CR2: 0000000000000000 ---[ end trace 59fc7d769346345d ]--- RIP: 0010:0x0 Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6. RSP: 0018:ffffc90002e47508 EFLAGS: 00010246 RAX: dffffc0000000000 RBX: 1ffff920005c8ea3 RCX: 0000000000000000 RDX: ffffc90002e47548 RSI: 0000000000000012 RDI: ffff888012b52000 RBP: ffff888012b52000 R08: ffff888019c30c18 R09: 0000000000000000 R10: ffffffff8979d8e8 R11: fffff520005c8ea1 R12: ffffffff890469c0 R13: 00000000ffff0000 R14: ffff888019c30c1c R15: ffff888019e7a000 FS: 00007f5bf96f2700(0000) GS:ffff8880b9f00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fff85574850 CR3: 00000000143db000 CR4: 00000000001506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400