syzbot


possible deadlock in sch_direct_xmit (2)

Status: upstream: reported C repro on 2024/01/09 18:28
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+834304373a84ac317e98@syzkaller.appspotmail.com
First crash: 158d, last: 32d
Fix commit to backport (bisect log) :
tree: upstream
commit 0bef512012b1cd8820f0c9ec80e5f8ceb43fdd59
Author: Eric Dumazet <edumazet@google.com>
Date: Mon Feb 12 14:07:00 2024 +0000

  net: add netdev_lockdep_set_classes() to virtual drivers

  
Bug presence (2)
Date Name Commit Repro Result
2024/04/26 linux-6.1.y (ToT) 6741e066ec76 C [report] possible deadlock in sch_direct_xmit
2024/04/26 upstream (ToT) c942a0cd3603 C Didn't crash
Similar bugs (12)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
android-44 possible deadlock in sch_direct_xmit C 240 1654d 1891d 0/2 public: reported C repro on 2019/04/11 08:44
upstream possible deadlock in sch_direct_xmit (2) net C done unreliable 109 333d 1508d 0/27 auto-obsoleted due to no activity on 2024/01/14 06:05
linux-4.19 possible deadlock in sch_direct_xmit (2) C error 15 475d 993d 0/1 upstream: reported C repro on 2021/09/26 01:30
upstream possible deadlock in sch_direct_xmit net C done done 1548 1662d 2341d 15/27 fixed on 2020/04/17 19:57
linux-5.15 possible deadlock in sch_direct_xmit (2) origin:lts-only C error 6 27d 113d 0/3 upstream: reported C repro on 2024/02/22 19:25
linux-4.14 possible deadlock in sch_direct_xmit 1 1839d 1838d 0/1 auto-closed as invalid on 2019/10/25 08:40
upstream possible deadlock in sch_direct_xmit (4) net 1 48d 48d 26/27 fixed on 2024/06/05 13:52
linux-4.14 possible deadlock in sch_direct_xmit (2) 1 1672d 1672d 0/1 auto-closed as invalid on 2020/03/15 19:58
linux-4.19 possible deadlock in sch_direct_xmit 1 1840d 1840d 0/1 auto-closed as invalid on 2019/10/25 08:50
linux-5.15 possible deadlock in sch_direct_xmit 1 400d 400d 0/3 auto-obsoleted due to no activity on 2023/08/23 09:09
linux-6.1 possible deadlock in sch_direct_xmit 2 408d 446d 0/3 auto-obsoleted due to no activity on 2023/08/23 09:10
upstream possible deadlock in sch_direct_xmit (3) net 1 123d 123d 26/27 fixed on 2024/04/10 16:40
Fix bisection attempts (1)
Created Duration User Patch Repo Result
2024/05/19 07:57 4h04m fix candidate upstream job log (1)

Sample crash report:
============================================
WARNING: possible recursive locking detected
6.1.85-syzkaller #0 Not tainted
--------------------------------------------
syz-executor871/4118 is trying to acquire lock:
ffff88807a065458 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff88807a065458 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4304 [inline]
ffff88807a065458 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c0/0x5e0 net/sched/sch_generic.c:340

but task is already holding lock:
ffff888019b6a0d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff888019b6a0d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4304 [inline]
ffff888019b6a0d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c0/0x5e0 net/sched/sch_generic.c:340

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(_xmit_ETHER#2);
  lock(_xmit_ETHER#2);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

12 locks held by syz-executor871/4118:
 #0: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:350 [inline]
 #0: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:791 [inline]
 #0: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: rawv6_send_hdrinc+0xa8b/0x1850 net/ipv6/raw.c:652
 #1: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:350 [inline]
 #1: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:791 [inline]
 #1: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: ip6_finish_output2+0x698/0x1520 net/ipv6/ip6_output.c:119
 #2: ffffffff8d12aca0 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
 #2: ffffffff8d12aca0 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:843 [inline]
 #2: ffffffff8d12aca0 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x2d6/0x3cf0 net/core/dev.c:4216
 #3: ffff888074787258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
 #3: ffff888074787258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:194 [inline]
 #3: ffff888074787258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3841 [inline]
 #3: ffff888074787258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x131b/0x3cf0 net/core/dev.c:4259
 #4: ffff888019b6a0d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
 #4: ffff888019b6a0d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4304 [inline]
 #4: ffff888019b6a0d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c0/0x5e0 net/sched/sch_generic.c:340
 #5: ffff8880281780b0 (k-slock-AF_INET6){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
 #5: ffff8880281780b0 (k-slock-AF_INET6){+...}-{2:2}, at: icmpv6_xmit_lock net/ipv6/icmp.c:108 [inline]
 #5: ffff8880281780b0 (k-slock-AF_INET6){+...}-{2:2}, at: icmp6_send+0xc10/0x2160 net/ipv6/icmp.c:543
 #6: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
 #6: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: icmp6_send+0xb70/0x2160 net/ipv6/icmp.c:535
 #7: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:350 [inline]
 #7: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:791 [inline]
 #7: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: ip6_finish_output2+0x698/0x1520 net/ipv6/ip6_output.c:119
 #8: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:350 [inline]
 #8: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:791 [inline]
 #8: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: ip6_nd_hdr net/ipv6/ndisc.c:452 [inline]
 #8: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: ndisc_send_skb+0x5a8/0x14e0 net/ipv6/ndisc.c:503
 #9: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:350 [inline]
 #9: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:791 [inline]
 #9: ffffffff8d12ac40 (rcu_read_lock){....}-{1:2}, at: ip6_finish_output2+0x698/0x1520 net/ipv6/ip6_output.c:119
 #10: ffffffff8d12aca0 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
 #10: ffffffff8d12aca0 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:843 [inline]
 #10: ffffffff8d12aca0 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x2d6/0x3cf0 net/core/dev.c:4216
 #11: ffff88807a4c5258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
 #11: ffff88807a4c5258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:194 [inline]
 #11: ffff88807a4c5258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3841 [inline]
 #11: ffff88807a4c5258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x131b/0x3cf0 net/core/dev.c:4259

stack backtrace:
CPU: 0 PID: 4118 Comm: syz-executor871 Not tainted 6.1.85-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2cb lib/dump_stack.c:106
 print_deadlock_bug kernel/locking/lockdep.c:2983 [inline]
 check_deadlock kernel/locking/lockdep.c:3026 [inline]
 validate_chain+0x4711/0x5950 kernel/locking/lockdep.c:3812
 __lock_acquire+0x125b/0x1f80 kernel/locking/lockdep.c:5049
 lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:351 [inline]
 __netif_tx_lock include/linux/netdevice.h:4304 [inline]
 sch_direct_xmit+0x1c0/0x5e0 net/sched/sch_generic.c:340
 __dev_xmit_skb net/core/dev.c:3854 [inline]
 __dev_queue_xmit+0x1ace/0x3cf0 net/core/dev.c:4259
 dev_queue_xmit include/linux/netdevice.h:3021 [inline]
 neigh_hh_output include/net/neighbour.h:528 [inline]
 neigh_output include/net/neighbour.h:542 [inline]
 ip6_finish_output2+0xea3/0x1520 net/ipv6/ip6_output.c:134
 __ip6_finish_output net/ipv6/ip6_output.c:201 [inline]
 ip6_finish_output+0x6a0/0xa80 net/ipv6/ip6_output.c:212
 dst_output include/net/dst.h:444 [inline]
 NF_HOOK include/linux/netfilter.h:302 [inline]
 ndisc_send_skb+0xbab/0x14e0 net/ipv6/ndisc.c:509
 ndisc_solicit+0x48a/0x690
 neigh_probe net/core/neighbour.c:1075 [inline]
 __neigh_event_send+0xf30/0x1610 net/core/neighbour.c:1242
 neigh_event_send_probe include/net/neighbour.h:468 [inline]
 neigh_event_send include/net/neighbour.h:474 [inline]
 neigh_resolve_output+0x1b0/0x710 net/core/neighbour.c:1547
 neigh_output include/net/neighbour.h:544 [inline]
 ip6_finish_output2+0xedb/0x1520 net/ipv6/ip6_output.c:134
 __ip6_finish_output net/ipv6/ip6_output.c:201 [inline]
 ip6_finish_output+0x6a0/0xa80 net/ipv6/ip6_output.c:212
 ip6_send_skb+0x12b/0x240 net/ipv6/ip6_output.c:1995
 icmp6_send+0x166f/0x2160 net/ipv6/icmp.c:622
 __icmpv6_send include/linux/icmpv6.h:28 [inline]
 icmpv6_send include/linux/icmpv6.h:49 [inline]
 ip6_link_failure+0x38/0x4e0 net/ipv6/route.c:2789
 ip_tunnel_xmit+0x1905/0x2cc0 net/ipv4/ip_tunnel.c:846
 __gre_xmit net/ipv4/ip_gre.c:474 [inline]
 erspan_xmit+0xaa2/0x1530 net/ipv4/ip_gre.c:723
 __netdev_start_xmit include/linux/netdevice.h:4853 [inline]
 netdev_start_xmit include/linux/netdevice.h:4867 [inline]
 xmit_one net/core/dev.c:3627 [inline]
 dev_hard_start_xmit+0x261/0x8c0 net/core/dev.c:3643
 sch_direct_xmit+0x2b2/0x5e0 net/sched/sch_generic.c:342
 __dev_xmit_skb net/core/dev.c:3854 [inline]
 __dev_queue_xmit+0x1ace/0x3cf0 net/core/dev.c:4259
 dev_queue_xmit include/linux/netdevice.h:3021 [inline]
 neigh_hh_output include/net/neighbour.h:528 [inline]
 neigh_output include/net/neighbour.h:542 [inline]
 ip6_finish_output2+0xea3/0x1520 net/ipv6/ip6_output.c:134
 __ip6_finish_output net/ipv6/ip6_output.c:201 [inline]
 ip6_finish_output+0x6a0/0xa80 net/ipv6/ip6_output.c:212
 dst_output include/net/dst.h:444 [inline]
 NF_HOOK include/linux/netfilter.h:302 [inline]
 rawv6_send_hdrinc+0xd15/0x1850 net/ipv6/raw.c:654
 rawv6_sendmsg+0x1648/0x2150 net/ipv6/raw.c:912
 sock_sendmsg_nosec net/socket.c:718 [inline]
 __sock_sendmsg net/socket.c:730 [inline]
 ____sys_sendmsg+0x5a5/0x8f0 net/socket.c:2514
 ___sys_sendmsg net/socket.c:2568 [inline]
 __sys_sendmmsg+0x3ab/0x730 net/socket.c:2654
 __do_sys_sendmmsg net/socket.c:2683 [inline]
 __se_sys_sendmmsg net/socket.c:2680 [inline]
 __x64_sys_sendmmsg+0x9c/0xb0 net/socket.c:2680
 do_syscall_x64 arch/x86/entry/common.c:51 [inline]
 do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:81
 entry_SYSCALL_64_after_hwframe+0x68/0xd2
RIP: 0033:0x7f65d7045369
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 d1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fffa1747ad8 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f65d7045369
RDX: 0000000000000002 RSI: 0000000020001e80 RDI: 0000000000000005
RBP: 0000000000000000 R08: 0000000000000000 R09: 00007fffa1747b70
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 </TASK>
syz-executor871 (4118) used greatest stack depth: 17848 bytes left

Crashes (11):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/04/11 00:34 linux-6.1.y bf1e3b1cb1e0 4320ec32 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in sch_direct_xmit
2024/04/10 16:42 linux-6.1.y bf1e3b1cb1e0 4320ec32 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in sch_direct_xmit
2024/04/07 08:14 linux-6.1.y 347385861c50 ca620dd8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in sch_direct_xmit
2024/04/03 15:44 linux-6.1.y 347385861c50 51c4dcff .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in sch_direct_xmit
2024/04/03 15:44 linux-6.1.y 347385861c50 51c4dcff .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in sch_direct_xmit
2024/03/20 16:27 linux-6.1.y d7543167affd 5b7d42ae .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in sch_direct_xmit
2024/03/09 10:20 linux-6.1.y 61adba85cc40 6ee49f2e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in sch_direct_xmit
2024/01/08 05:12 linux-6.1.y 38fb82ecd144 d0304e9c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in sch_direct_xmit
2024/05/13 02:40 linux-6.1.y 909ba1f1b414 9026e142 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in sch_direct_xmit
2024/05/01 04:35 linux-6.1.y dcbc050cb0d3 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in sch_direct_xmit
2024/01/24 15:59 linux-6.1.y 8fd7f4462453 1e153dc8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in sch_direct_xmit
* Struck through repros no longer work on HEAD.