syzbot

 sign-in | mailing list | source | docs
🐞 Open [1278]
≡ Subsystems
🐞 Fixed [5716]
🐞 Invalid [13928]
⬇ Missing Backports [97]
Kernel Health Bugs/Month Bug Lifetimes Fuzzing Crashes
Total Repo Heatmap Subsystems Heatmap
💬 Send us feedback

possible deadlock in sch_direct_xmit (4)

Status: fixed on 2024/06/05 13:52
Subsystems: net
[Documentation on labels]
Fix commit: 86735b57c905 net/sched: unregister lockdep keys in qdisc_create/qdisc_alloc error path
First crash: 166d, last: 166d
▶ ▼ Similar bugs (12)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 possible deadlock in sch_direct_xmit (2) origin:lts-only C done 17 9d05h 275d 0/3 upstream: reported C repro on 2024/01/09 18:28
android-44 possible deadlock in sch_direct_xmit C 240 1773d 2010d 0/2 public: reported C repro on 2019/04/11 08:44
upstream possible deadlock in sch_direct_xmit (2) net C done unreliable 109 452d 1626d 0/28 auto-obsoleted due to no activity on 2024/01/14 06:05
linux-4.19 possible deadlock in sch_direct_xmit (2) C error 15 594d 1111d 0/1 upstream: reported C repro on 2021/09/26 01:30
upstream possible deadlock in sch_direct_xmit net C done done 1548 1781d 2460d 15/28 fixed on 2020/04/17 19:57
linux-5.15 possible deadlock in sch_direct_xmit (2) origin:lts-only C error 12 3d06h 231d 0/3 upstream: reported C repro on 2024/02/22 19:25
linux-4.14 possible deadlock in sch_direct_xmit 1 1957d 1957d 0/1 auto-closed as invalid on 2019/10/25 08:40
linux-4.14 possible deadlock in sch_direct_xmit (2) 1 1790d 1790d 0/1 auto-closed as invalid on 2020/03/15 19:58
linux-4.19 possible deadlock in sch_direct_xmit 1 1959d 1959d 0/1 auto-closed as invalid on 2019/10/25 08:50
linux-5.15 possible deadlock in sch_direct_xmit 1 519d 519d 0/3 auto-obsoleted due to no activity on 2023/08/23 09:09
linux-6.1 possible deadlock in sch_direct_xmit 2 527d 565d 0/3 auto-obsoleted due to no activity on 2023/08/23 09:10
upstream possible deadlock in sch_direct_xmit (3) net 1 242d 242d 25/28 fixed on 2024/04/10 16:40

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.9.0-rc5-syzkaller-00171-gb2ff42c6d3ab #0 Not tainted
------------------------------------------------------
syz-executor.4/18511 is trying to acquire lock:
ffff88807db3d4d8 (&qdisc_xmit_lock_key#3){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff88807db3d4d8 (&qdisc_xmit_lock_key#3){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4371 [inline]
ffff88807db3d4d8 (&qdisc_xmit_lock_key#3){+.-.}-{2:2}, at: sch_direct_xmit+0x1c4/0x5f0 net/sched/sch_generic.c:341

but task is already holding lock:
ffff88801e3488d8 (&qdisc_xmit_lock_key#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff88801e3488d8 (&qdisc_xmit_lock_key#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4371 [inline]
ffff88801e3488d8 (&qdisc_xmit_lock_key#2){+.-.}-{2:2}, at: __dev_queue_xmit+0x179d/0x3ca0 net/core/dev.c:4337

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&qdisc_xmit_lock_key#2){+.-.}-{2:2}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
       spin_lock include/linux/spinlock.h:351 [inline]
       __netif_tx_lock include/linux/netdevice.h:4371 [inline]
       __dev_queue_xmit+0x179d/0x3ca0 net/core/dev.c:4337
       neigh_output include/net/neighbour.h:542 [inline]
       ip_finish_output2+0xe70/0x1380 net/ipv4/ip_output.c:235
       iptunnel_xmit+0x540/0x9b0 net/ipv4/ip_tunnel_core.c:82
       ip_tunnel_xmit+0x20ef/0x2940 net/ipv4/ip_tunnel.c:857
       erspan_xmit+0xa18/0x1500 net/ipv4/ip_gre.c:725
       __netdev_start_xmit include/linux/netdevice.h:4903 [inline]
       netdev_start_xmit include/linux/netdevice.h:4917 [inline]
       xmit_one net/core/dev.c:3531 [inline]
       dev_hard_start_xmit+0x27a/0x7e0 net/core/dev.c:3547
       sch_direct_xmit+0x2b6/0x5f0 net/sched/sch_generic.c:343
       __dev_xmit_skb net/core/dev.c:3760 [inline]
       __dev_queue_xmit+0x19eb/0x3ca0 net/core/dev.c:4307
       neigh_output include/net/neighbour.h:542 [inline]
       ip_finish_output2+0xe70/0x1380 net/ipv4/ip_output.c:235
       ip_local_out net/ipv4/ip_output.c:129 [inline]
       ip_send_skb+0x74/0x100 net/ipv4/ip_output.c:1492
       udp_send_skb+0xaa4/0x1470 net/ipv4/udp.c:968
       udp_sendmsg+0x1c21/0x2a60 net/ipv4/udp.c:1256
       udpv6_sendmsg+0x1383/0x3270 net/ipv6/udp.c:1388
       sock_sendmsg_nosec net/socket.c:730 [inline]
       __sock_sendmsg+0xef/0x270 net/socket.c:745
       ____sys_sendmsg+0x525/0x7d0 net/socket.c:2584
       ___sys_sendmsg net/socket.c:2638 [inline]
       __sys_sendmmsg+0x3b2/0x740 net/socket.c:2724
       __do_sys_sendmmsg net/socket.c:2753 [inline]
       __se_sys_sendmmsg net/socket.c:2750 [inline]
       __x64_sys_sendmmsg+0xa0/0xb0 net/socket.c:2750
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #0 (&qdisc_xmit_lock_key#3){+.-.}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3134 [inline]
       check_prevs_add kernel/locking/lockdep.c:3253 [inline]
       validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
       __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
       spin_lock include/linux/spinlock.h:351 [inline]
       __netif_tx_lock include/linux/netdevice.h:4371 [inline]
       sch_direct_xmit+0x1c4/0x5f0 net/sched/sch_generic.c:341
       __dev_xmit_skb net/core/dev.c:3760 [inline]
       __dev_queue_xmit+0x19eb/0x3ca0 net/core/dev.c:4307
       dev_queue_xmit include/linux/netdevice.h:3091 [inline]
       neigh_hh_output include/net/neighbour.h:526 [inline]
       neigh_output include/net/neighbour.h:540 [inline]
       ip6_finish_output2+0xfc0/0x1670 net/ipv6/ip6_output.c:137
       ip6_finish_output+0x41e/0x810 net/ipv6/ip6_output.c:222
       NF_HOOK include/linux/netfilter.h:314 [inline]
       ndisc_send_skb+0xab0/0x1380 net/ipv6/ndisc.c:509
       ndisc_solicit+0x493/0x6a0
       neigh_probe net/core/neighbour.c:1075 [inline]
       __neigh_event_send+0xec8/0x15a0 net/core/neighbour.c:1242
       neigh_event_send_probe include/net/neighbour.h:466 [inline]
       neigh_event_send include/net/neighbour.h:472 [inline]
       neigh_resolve_output+0x1b5/0x740 net/core/neighbour.c:1547
       neigh_output include/net/neighbour.h:542 [inline]
       ip6_finish_output2+0xff8/0x1670 net/ipv6/ip6_output.c:137
       ip6_finish_output+0x41e/0x810 net/ipv6/ip6_output.c:222
       ip6_send_skb+0x112/0x230 net/ipv6/ip6_output.c:1955
       icmp6_send+0x15fc/0x2070 net/ipv6/icmp.c:624
       __icmpv6_send include/linux/icmpv6.h:28 [inline]
       icmpv6_send include/linux/icmpv6.h:49 [inline]
       ip6_link_failure+0x3c/0x4f0 net/ipv6/route.c:2797
       dst_link_failure include/net/dst.h:429 [inline]
       vti6_xmit net/ipv6/ip6_vti.c:547 [inline]
       vti6_tnl_xmit+0x13be/0x1bd0 net/ipv6/ip6_vti.c:585
       __netdev_start_xmit include/linux/netdevice.h:4903 [inline]
       netdev_start_xmit include/linux/netdevice.h:4917 [inline]
       xmit_one net/core/dev.c:3531 [inline]
       dev_hard_start_xmit+0x27a/0x7e0 net/core/dev.c:3547
       __dev_queue_xmit+0x1ad1/0x3ca0 net/core/dev.c:4341
       packet_snd net/packet/af_packet.c:3081 [inline]
       packet_sendmsg+0x46a9/0x6130 net/packet/af_packet.c:3113
       sock_sendmsg_nosec net/socket.c:730 [inline]
       __sock_sendmsg+0x221/0x270 net/socket.c:745
       __sys_sendto+0x3a4/0x4f0 net/socket.c:2191
       __do_sys_sendto net/socket.c:2203 [inline]
       __se_sys_sendto net/socket.c:2199 [inline]
       __x64_sys_sendto+0xde/0x100 net/socket.c:2199
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&qdisc_xmit_lock_key#2);
                               lock(&qdisc_xmit_lock_key#3);
                               lock(&qdisc_xmit_lock_key#2);
  lock(&qdisc_xmit_lock_key#3);

 *** DEADLOCK ***

9 locks held by syz-executor.4/18511:
 #0: ffffffff8e334d80 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
 #0: ffffffff8e334d80 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:833 [inline]
 #0: ffffffff8e334d80 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x2c7/0x3ca0 net/core/dev.c:4266
 #1: ffff88801e3488d8 (&qdisc_xmit_lock_key#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
 #1: ffff88801e3488d8 (&qdisc_xmit_lock_key#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4371 [inline]
 #1: ffff88801e3488d8 (&qdisc_xmit_lock_key#2){+.-.}-{2:2}, at: __dev_queue_xmit+0x179d/0x3ca0 net/core/dev.c:4337
 #2: ffff88802a6a01d8 (k-slock-AF_INET6){+.-.}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
 #2: ffff88802a6a01d8 (k-slock-AF_INET6){+.-.}-{2:2}, at: icmpv6_xmit_lock net/ipv6/icmp.c:108 [inline]
 #2: ffff88802a6a01d8 (k-slock-AF_INET6){+.-.}-{2:2}, at: icmp6_send+0xc45/0x2070 net/ipv6/icmp.c:545
 #3: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
 #3: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: icmp6_send+0xba4/0x2070 net/ipv6/icmp.c:537
 #4: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:329 [inline]
 #4: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:781 [inline]
 #4: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: ip6_finish_output2+0x712/0x1670 net/ipv6/ip6_output.c:122
 #5: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:329 [inline]
 #5: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:781 [inline]
 #5: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: ip6_nd_hdr net/ipv6/ndisc.c:452 [inline]
 #5: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: ndisc_send_skb+0x572/0x1380 net/ipv6/ndisc.c:503
 #6: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:329 [inline]
 #6: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:781 [inline]
 #6: ffffffff8e334d20 (rcu_read_lock){....}-{1:2}, at: ip6_finish_output2+0x712/0x1670 net/ipv6/ip6_output.c:122
 #7: ffffffff8e334d80 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
 #7: ffffffff8e334d80 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:833 [inline]
 #7: ffffffff8e334d80 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x2c7/0x3ca0 net/core/dev.c:4266
 #8: ffff88805e8fc258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock#2){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
 #8: ffff88805e8fc258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock#2){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:196 [inline]
 #8: ffff88805e8fc258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock#2){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3747 [inline]
 #8: ffff88805e8fc258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock#2){+...}-{2:2}, at: __dev_queue_xmit+0x1286/0x3ca0 net/core/dev.c:4307

stack backtrace:
CPU: 0 PID: 18511 Comm: syz-executor.4 Not tainted 6.9.0-rc5-syzkaller-00171-gb2ff42c6d3ab #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
 check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
 check_prev_add kernel/locking/lockdep.c:3134 [inline]
 check_prevs_add kernel/locking/lockdep.c:3253 [inline]
 validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
 __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:351 [inline]
 __netif_tx_lock include/linux/netdevice.h:4371 [inline]
 sch_direct_xmit+0x1c4/0x5f0 net/sched/sch_generic.c:341
 __dev_xmit_skb net/core/dev.c:3760 [inline]
 __dev_queue_xmit+0x19eb/0x3ca0 net/core/dev.c:4307
 dev_queue_xmit include/linux/netdevice.h:3091 [inline]
 neigh_hh_output include/net/neighbour.h:526 [inline]
 neigh_output include/net/neighbour.h:540 [inline]
 ip6_finish_output2+0xfc0/0x1670 net/ipv6/ip6_output.c:137
 ip6_finish_output+0x41e/0x810 net/ipv6/ip6_output.c:222
 NF_HOOK include/linux/netfilter.h:314 [inline]
 ndisc_send_skb+0xab0/0x1380 net/ipv6/ndisc.c:509
 ndisc_solicit+0x493/0x6a0
 neigh_probe net/core/neighbour.c:1075 [inline]
 __neigh_event_send+0xec8/0x15a0 net/core/neighbour.c:1242
 neigh_event_send_probe include/net/neighbour.h:466 [inline]
 neigh_event_send include/net/neighbour.h:472 [inline]
 neigh_resolve_output+0x1b5/0x740 net/core/neighbour.c:1547
 neigh_output include/net/neighbour.h:542 [inline]
 ip6_finish_output2+0xff8/0x1670 net/ipv6/ip6_output.c:137
 ip6_finish_output+0x41e/0x810 net/ipv6/ip6_output.c:222
 ip6_send_skb+0x112/0x230 net/ipv6/ip6_output.c:1955
 icmp6_send+0x15fc/0x2070 net/ipv6/icmp.c:624
 __icmpv6_send include/linux/icmpv6.h:28 [inline]
 icmpv6_send include/linux/icmpv6.h:49 [inline]
 ip6_link_failure+0x3c/0x4f0 net/ipv6/route.c:2797
 dst_link_failure include/net/dst.h:429 [inline]
 vti6_xmit net/ipv6/ip6_vti.c:547 [inline]
 vti6_tnl_xmit+0x13be/0x1bd0 net/ipv6/ip6_vti.c:585
 __netdev_start_xmit include/linux/netdevice.h:4903 [inline]
 netdev_start_xmit include/linux/netdevice.h:4917 [inline]
 xmit_one net/core/dev.c:3531 [inline]
 dev_hard_start_xmit+0x27a/0x7e0 net/core/dev.c:3547
 __dev_queue_xmit+0x1ad1/0x3ca0 net/core/dev.c:4341
 packet_snd net/packet/af_packet.c:3081 [inline]
 packet_sendmsg+0x46a9/0x6130 net/packet/af_packet.c:3113
 sock_sendmsg_nosec net/socket.c:730 [inline]
 __sock_sendmsg+0x221/0x270 net/socket.c:745
 __sys_sendto+0x3a4/0x4f0 net/socket.c:2191
 __do_sys_sendto net/socket.c:2203 [inline]
 __se_sys_sendto net/socket.c:2199 [inline]
 __x64_sys_sendto+0xde/0x100 net/socket.c:2199
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fe13b87dea9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fe13c51d0c8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
RAX: ffffffffffffffda RBX: 00007fe13b9abf80 RCX: 00007fe13b87dea9
RDX: 0000000000000028 RSI: 0000000020000100 RDI: 0000000000000005
RBP: 00007fe13b8ca4a4 R08: 0000000020000240 R09: 0000000000000014
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007fe13b9abf80 R15: 00007ffc3445c7f8
 </TASK>
netlink: 248 bytes leftover after parsing attributes in process `syz-executor.4'.

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/04/27 22:02 net b2ff42c6d3ab 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce possible deadlock in sch_direct_xmit
* Struck through repros no longer work on HEAD.