syzbot

 sign-in | mailing list | source | docs
🐞 Open [717] 🐞 Fixed [181] 🐞 Invalid [640] 📈 Kernel Health 📈 Bug Lifetimes 📈 Fuzzing 📈 Crashes 💬 Send us feedback

possible deadlock in __dev_queue_xmit

Status: auto-closed as invalid on 2021/01/16 21:33
Reported-by: syzbot+613ff5e2182ed06b7af0@syzkaller.appspotmail.com
First crash: 1774d, last: 1329d
▶ ▼ Similar bugs (11)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
android-49 possible deadlock in __dev_queue_xmit 4 1784d 1853d 0/3 auto-closed as invalid on 2019/10/25 08:36
upstream possible deadlock in __dev_queue_xmit (3) net C done inconclusive 986 5d18h 1620d 0/26 upstream: reported C repro on 2019/12/03 09:55
linux-4.19 possible deadlock in __dev_queue_xmit C error 5 961d 1379d 0/1 upstream: reported C repro on 2020/07/31 07:05
android-414 possible deadlock in __dev_queue_xmit 3 1842d 1853d 0/1 auto-closed as invalid on 2019/10/21 21:31
android-44 possible deadlock in __dev_queue_xmit 14 1619d 1681d 0/2 auto-closed as invalid on 2020/04/02 07:14
linux-6.1 possible deadlock in __dev_queue_xmit 98 203d 317d 0/3 auto-obsoleted due to no activity on 2023/12/28 22:22
upstream possible deadlock in __dev_queue_xmit net 1 1942d 1942d 0/26 closed as invalid on 2019/03/10 18:51
linux-5.15 possible deadlock in __dev_queue_xmit 113 268d 404d 0/3 auto-obsoleted due to no activity on 2023/10/25 04:56
linux-6.1 possible deadlock in __dev_queue_xmit (2) 7 2d11h 112d 0/3 upstream: reported on 2024/01/19 16:24
linux-5.15 possible deadlock in __dev_queue_xmit (2) origin:lts-only C 12 15d 147d 0/3 upstream: reported C repro on 2023/12/15 02:11
upstream possible deadlock in __dev_queue_xmit (2) kernel 2 1754d 1870d 0/26 auto-closed as invalid on 2019/11/19 09:01

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
4.14.198-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.3/10446 is trying to acquire lock:
 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}, at: [<ffffffff85146685>] spin_lock include/linux/spinlock.h:317 [inline]
 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}, at: [<ffffffff85146685>] __dev_xmit_skb net/core/dev.c:3198 [inline]
 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}, at: [<ffffffff85146685>] __dev_queue_xmit+0x1ef5/0x2480 net/core/dev.c:3489

but task is already holding lock:
 (_xmit_ETHER#2){+.-.}, at: [<ffffffff8521e10b>] spin_lock include/linux/spinlock.h:317 [inline]
 (_xmit_ETHER#2){+.-.}, at: [<ffffffff8521e10b>] __netif_tx_lock include/linux/netdevice.h:3536 [inline]
 (_xmit_ETHER#2){+.-.}, at: [<ffffffff8521e10b>] sch_direct_xmit+0x1db/0x500 net/sched/sch_generic.c:184

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (_xmit_ETHER#2){+.-.}:
       __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:152
       spin_lock include/linux/spinlock.h:317 [inline]
       __netif_tx_lock include/linux/netdevice.h:3536 [inline]
       sch_direct_xmit+0x1db/0x500 net/sched/sch_generic.c:184
       __dev_xmit_skb net/core/dev.c:3214 [inline]
       __dev_queue_xmit+0x1ab0/0x2480 net/core/dev.c:3489
       neigh_hh_output include/net/neighbour.h:490 [inline]
       neigh_output include/net/neighbour.h:498 [inline]
       ip_finish_output2+0x9db/0x12f0 net/ipv4/ip_output.c:229
       ip_finish_output+0x37b/0xc30 net/ipv4/ip_output.c:317
       NF_HOOK_COND include/linux/netfilter.h:239 [inline]
       ip_mc_output+0x220/0xcb0 net/ipv4/ip_output.c:390
       dst_output include/net/dst.h:470 [inline]
       ip_local_out+0x93/0x170 net/ipv4/ip_output.c:124
       ip_send_skb+0x3a/0xc0 net/ipv4/ip_output.c:1422
       udp_send_skb+0x601/0xb70 net/ipv4/udp.c:833
       udp_sendmsg+0x15a3/0x1c00 net/ipv4/udp.c:1057
       inet_sendmsg+0x11a/0x4e0 net/ipv4/af_inet.c:762
       sock_sendmsg_nosec net/socket.c:646 [inline]
       sock_sendmsg+0xb5/0x100 net/socket.c:656
       ___sys_sendmsg+0x326/0x800 net/socket.c:2062
       __sys_sendmmsg+0x129/0x330 net/socket.c:2152
       SYSC_sendmmsg net/socket.c:2183 [inline]
       SyS_sendmmsg+0x2f/0x50 net/socket.c:2178
       do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x46/0xbb

-> #0 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}:
       lock_acquire+0x170/0x3f0 kernel/locking/lockdep.c:3998
       __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:152
       spin_lock include/linux/spinlock.h:317 [inline]
       __dev_xmit_skb net/core/dev.c:3198 [inline]
       __dev_queue_xmit+0x1ef5/0x2480 net/core/dev.c:3489
       neigh_hh_output include/net/neighbour.h:490 [inline]
       neigh_output include/net/neighbour.h:498 [inline]
       ip_finish_output2+0x9db/0x12f0 net/ipv4/ip_output.c:229
       ip_finish_output+0x37b/0xc30 net/ipv4/ip_output.c:317
       NF_HOOK_COND include/linux/netfilter.h:239 [inline]
       ip_mc_output+0x220/0xcb0 net/ipv4/ip_output.c:390
       dst_output include/net/dst.h:470 [inline]
       ip_local_out+0x93/0x170 net/ipv4/ip_output.c:124
       iptunnel_xmit+0x5cc/0x950 net/ipv4/ip_tunnel_core.c:91
       ip_tunnel_xmit+0xef8/0x3440 net/ipv4/ip_tunnel.c:799
       erspan_xmit net/ipv4/ip_gre.c:750 [inline]
       erspan_xmit+0x43e/0x1090 net/ipv4/ip_gre.c:725
       __netdev_start_xmit include/linux/netdevice.h:4039 [inline]
       netdev_start_xmit include/linux/netdevice.h:4048 [inline]
       xmit_one net/core/dev.c:3005 [inline]
       dev_hard_start_xmit+0x188/0x890 net/core/dev.c:3021
       sch_direct_xmit+0x251/0x500 net/sched/sch_generic.c:186
       __dev_xmit_skb net/core/dev.c:3214 [inline]
       __dev_queue_xmit+0x1ab0/0x2480 net/core/dev.c:3489
       neigh_hh_output include/net/neighbour.h:490 [inline]
       neigh_output include/net/neighbour.h:498 [inline]
       ip_finish_output2+0x9db/0x12f0 net/ipv4/ip_output.c:229
       ip_finish_output+0x37b/0xc30 net/ipv4/ip_output.c:317
       NF_HOOK_COND include/linux/netfilter.h:239 [inline]
       ip_mc_output+0x220/0xcb0 net/ipv4/ip_output.c:390
       dst_output include/net/dst.h:470 [inline]
       ip_local_out+0x93/0x170 net/ipv4/ip_output.c:124
       ip_send_skb+0x3a/0xc0 net/ipv4/ip_output.c:1422
       udp_send_skb+0x601/0xb70 net/ipv4/udp.c:833
       udp_sendmsg+0x15a3/0x1c00 net/ipv4/udp.c:1057
       inet_sendmsg+0x11a/0x4e0 net/ipv4/af_inet.c:762
       sock_sendmsg_nosec net/socket.c:646 [inline]
       sock_sendmsg+0xb5/0x100 net/socket.c:656
       ___sys_sendmsg+0x326/0x800 net/socket.c:2062
       __sys_sendmmsg+0x129/0x330 net/socket.c:2152
       SYSC_sendmmsg net/socket.c:2183 [inline]
       SyS_sendmmsg+0x2f/0x50 net/socket.c:2178
       do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x46/0xbb

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(_xmit_ETHER#2);
                               lock(dev->qdisc_tx_busylock ?: &qdisc_tx_busylock);
                               lock(_xmit_ETHER#2);
  lock(dev->qdisc_tx_busylock ?: &qdisc_tx_busylock);

 *** DEADLOCK ***

6 locks held by syz-executor.3/10446:
 #0:  (rcu_read_lock_bh){....}, at: [<ffffffff855d9acf>] lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #0:  (rcu_read_lock_bh){....}, at: [<ffffffff855d9acf>] ip_finish_output2+0x23f/0x12f0 net/ipv4/ip_output.c:213
 #1:  (rcu_read_lock_bh){....}, at: [<ffffffff8514493c>] __dev_queue_xmit+0x1ac/0x2480 net/core/dev.c:3455
 #2:  (dev->qdisc_running_key ?: &qdisc_running_key){+...}, at: [<ffffffff855da26b>] neigh_hh_output include/net/neighbour.h:490 [inline]
 #2:  (dev->qdisc_running_key ?: &qdisc_running_key){+...}, at: [<ffffffff855da26b>] neigh_output include/net/neighbour.h:498 [inline]
 #2:  (dev->qdisc_running_key ?: &qdisc_running_key){+...}, at: [<ffffffff855da26b>] ip_finish_output2+0x9db/0x12f0 net/ipv4/ip_output.c:229
 #3:  (_xmit_ETHER#2){+.-.}, at: [<ffffffff8521e10b>] spin_lock include/linux/spinlock.h:317 [inline]
 #3:  (_xmit_ETHER#2){+.-.}, at: [<ffffffff8521e10b>] __netif_tx_lock include/linux/netdevice.h:3536 [inline]
 #3:  (_xmit_ETHER#2){+.-.}, at: [<ffffffff8521e10b>] sch_direct_xmit+0x1db/0x500 net/sched/sch_generic.c:184
 #4:  (rcu_read_lock_bh){....}, at: [<ffffffff855d9acf>] lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #4:  (rcu_read_lock_bh){....}, at: [<ffffffff855d9acf>] ip_finish_output2+0x23f/0x12f0 net/ipv4/ip_output.c:213
 #5:  (rcu_read_lock_bh){....}, at: [<ffffffff8514493c>] __dev_queue_xmit+0x1ac/0x2480 net/core/dev.c:3455

stack backtrace:
CPU: 0 PID: 10446 Comm: syz-executor.3 Not tainted 4.14.198-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x1b2/0x283 lib/dump_stack.c:58
 print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1258
 check_prev_add kernel/locking/lockdep.c:1905 [inline]
 check_prevs_add kernel/locking/lockdep.c:2022 [inline]
 validate_chain kernel/locking/lockdep.c:2464 [inline]
 __lock_acquire+0x2e0e/0x3f20 kernel/locking/lockdep.c:3491
 lock_acquire+0x170/0x3f0 kernel/locking/lockdep.c:3998
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:152
 spin_lock include/linux/spinlock.h:317 [inline]
 __dev_xmit_skb net/core/dev.c:3198 [inline]
 __dev_queue_xmit+0x1ef5/0x2480 net/core/dev.c:3489
 neigh_hh_output include/net/neighbour.h:490 [inline]
 neigh_output include/net/neighbour.h:498 [inline]
 ip_finish_output2+0x9db/0x12f0 net/ipv4/ip_output.c:229
 ip_finish_output+0x37b/0xc30 net/ipv4/ip_output.c:317
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip_mc_output+0x220/0xcb0 net/ipv4/ip_output.c:390
 dst_output include/net/dst.h:470 [inline]
 ip_local_out+0x93/0x170 net/ipv4/ip_output.c:124
 iptunnel_xmit+0x5cc/0x950 net/ipv4/ip_tunnel_core.c:91
 ip_tunnel_xmit+0xef8/0x3440 net/ipv4/ip_tunnel.c:799
 erspan_xmit net/ipv4/ip_gre.c:750 [inline]
 erspan_xmit+0x43e/0x1090 net/ipv4/ip_gre.c:725
 __netdev_start_xmit include/linux/netdevice.h:4039 [inline]
 netdev_start_xmit include/linux/netdevice.h:4048 [inline]
 xmit_one net/core/dev.c:3005 [inline]
 dev_hard_start_xmit+0x188/0x890 net/core/dev.c:3021
 sch_direct_xmit+0x251/0x500 net/sched/sch_generic.c:186
 __dev_xmit_skb net/core/dev.c:3214 [inline]
 __dev_queue_xmit+0x1ab0/0x2480 net/core/dev.c:3489
 neigh_hh_output include/net/neighbour.h:490 [inline]
 neigh_output include/net/neighbour.h:498 [inline]
 ip_finish_output2+0x9db/0x12f0 net/ipv4/ip_output.c:229
 ip_finish_output+0x37b/0xc30 net/ipv4/ip_output.c:317
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip_mc_output+0x220/0xcb0 net/ipv4/ip_output.c:390
 dst_output include/net/dst.h:470 [inline]
 ip_local_out+0x93/0x170 net/ipv4/ip_output.c:124
 ip_send_skb+0x3a/0xc0 net/ipv4/ip_output.c:1422
 udp_send_skb+0x601/0xb70 net/ipv4/udp.c:833
 udp_sendmsg+0x15a3/0x1c00 net/ipv4/udp.c:1057
 inet_sendmsg+0x11a/0x4e0 net/ipv4/af_inet.c:762
 sock_sendmsg_nosec net/socket.c:646 [inline]
 sock_sendmsg+0xb5/0x100 net/socket.c:656
 ___sys_sendmsg+0x326/0x800 net/socket.c:2062
 __sys_sendmmsg+0x129/0x330 net/socket.c:2152
 SYSC_sendmmsg net/socket.c:2183 [inline]
 SyS_sendmmsg+0x2f/0x50 net/socket.c:2178
 do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x46/0xbb
RIP: 0033:0x45d5f9
RSP: 002b:00007ff386ce3c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 0000000000027a40 RCX: 000000000045d5f9
RDX: 000000000800001d RSI: 0000000020007fc0 RDI: 0000000000000003
RBP: 000000000118d028 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000118cfec
R13: 00007ffd89fd96af R14: 00007ff386ce49c0 R15: 000000000118cfec
ptrace attach of "/root/syz-executor.4"[10504] was attempted by "/root/syz-executor.4"[10506]
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.4'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.4'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.4'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.4'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.0'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.4'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.0'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.4'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.0'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.4'.
kvm [10701]: vcpu0, guest rIP: 0x14c disabled perfctr wrmsr: 0xc2 data 0x65
kvm [10701]: vcpu0, guest rIP: 0x14c disabled perfctr wrmsr: 0xc2 data 0x64
kvm [10701]: vcpu0, guest rIP: 0x14c disabled perfctr wrmsr: 0xc2 data 0x65
kvm [10701]: vcpu0, guest rIP: 0x14c disabled perfctr wrmsr: 0xc2 data 0x64
kvm [10701]: vcpu0, guest rIP: 0x14c disabled perfctr wrmsr: 0xc2 data 0x65
kvm [10701]: vcpu0, guest rIP: 0x14c disabled perfctr wrmsr: 0xc2 data 0x64
kvm [10701]: vcpu0, guest rIP: 0x14c disabled perfctr wrmsr: 0xc2 data 0x65
kvm [10701]: vcpu0, guest rIP: 0x14c disabled perfctr wrmsr: 0xc2 data 0x64
kvm [10701]: vcpu0, guest rIP: 0x14c disabled perfctr wrmsr: 0xc2 data 0x65
kvm [10701]: vcpu0, guest rIP: 0x14c disabled perfctr wrmsr: 0xc2 data 0x64

Crashes (7):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/09/18 21:33 linux-4.14.y cbfa1702aaf6 53ce8104 .config console log report info ci2-linux-4-14
2020/06/10 07:30 linux-4.14.y c6db52a88798 860c4de9 .config console log report ci2-linux-4-14
2020/04/21 22:20 linux-4.14.y c10b57a567e4 f20434a8 .config console log report ci2-linux-4-14
2020/02/09 22:03 linux-4.14.y e0f8b8a65a47 35f5e45e .config console log report ci2-linux-4-14
2019/11/21 22:22 linux-4.14.y f56f3d0e65ad 8098ea0f .config console log report ci2-linux-4-14
2019/10/12 14:29 linux-4.14.y e132c8d7b58d 426631dd .config console log report ci2-linux-4-14
2019/07/01 23:38 linux-4.14.y f4cc0ed9b2c7 cccc4302 .config console log report ci2-linux-4-14
* Struck through repros no longer work on HEAD.