syzbot


possible deadlock in __dev_queue_xmit (2)

Status: upstream: reported C repro on 2023/12/15 02:11
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+3f83cf7d857e86ce7d54@syzkaller.appspotmail.com
First crash: 139d, last: 7d14h
Bug presence (2)
Date Name Commit Repro Result
2024/04/25 linux-5.15.y (ToT) c52b9710c83d C [report] possible deadlock in __dev_queue_xmit
2024/04/25 upstream (ToT) e88c4cfcb7b8 C Didn't crash
Similar bugs (11)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
android-49 possible deadlock in __dev_queue_xmit 4 1776d 1845d 0/3 auto-closed as invalid on 2019/10/25 08:36
upstream possible deadlock in __dev_queue_xmit (3) net C done inconclusive 984 24d 1612d 0/26 upstream: reported C repro on 2019/12/03 09:55
linux-4.19 possible deadlock in __dev_queue_xmit C error 5 953d 1371d 0/1 upstream: reported C repro on 2020/07/31 07:05
android-414 possible deadlock in __dev_queue_xmit 3 1835d 1846d 0/1 auto-closed as invalid on 2019/10/21 21:31
android-44 possible deadlock in __dev_queue_xmit 14 1611d 1673d 0/2 auto-closed as invalid on 2020/04/02 07:14
linux-6.1 possible deadlock in __dev_queue_xmit 98 196d 309d 0/3 auto-obsoleted due to no activity on 2023/12/28 22:22
linux-4.14 possible deadlock in __dev_queue_xmit 7 1322d 1766d 0/1 auto-closed as invalid on 2021/01/16 21:33
upstream possible deadlock in __dev_queue_xmit net 1 1934d 1934d 0/26 closed as invalid on 2019/03/10 18:51
linux-5.15 possible deadlock in __dev_queue_xmit 113 260d 396d 0/3 auto-obsoleted due to no activity on 2023/10/25 04:56
linux-6.1 possible deadlock in __dev_queue_xmit (2) 6 20d 104d 0/3 upstream: reported on 2024/01/19 16:24
upstream possible deadlock in __dev_queue_xmit (2) kernel 2 1746d 1862d 0/26 auto-closed as invalid on 2019/11/19 09:01

Sample crash report:
============================================
WARNING: possible recursive locking detected
5.15.153-syzkaller #0 Not tainted
--------------------------------------------
syz-executor427/3843 is trying to acquire lock:
ffff88801fa8f498 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:363 [inline]
ffff88801fa8f498 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4429 [inline]
ffff88801fa8f498 (_xmit_ETHER#2){+.-.}-{2:2}, at: __dev_queue_xmit+0x15be/0x3230 net/core/dev.c:4244

but task is already holding lock:
ffff888023b98498 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:363 [inline]
ffff888023b98498 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4429 [inline]
ffff888023b98498 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c0/0x5e0 net/sched/sch_generic.c:340

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(_xmit_ETHER#2);
  lock(_xmit_ETHER#2);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

9 locks held by syz-executor427/3843:
 #0: ffffffff8c91f720 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x5/0x30 include/linux/rcupdate.h:311
 #1: ffffffff8c91f780 (rcu_read_lock_bh){....}-{1:2}, at: rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:312
 #2: ffffffff8c91f780 (rcu_read_lock_bh){....}-{1:2}, at: rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:312
 #3: ffff888077e44258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:373 [inline]
 #3: ffff888077e44258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:173 [inline]
 #3: ffff888077e44258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3831 [inline]
 #3: ffff888077e44258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x11f2/0x3230 net/core/dev.c:4213
 #4: ffff888023b98498 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:363 [inline]
 #4: ffff888023b98498 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4429 [inline]
 #4: ffff888023b98498 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c0/0x5e0 net/sched/sch_generic.c:340
 #5: ffff888023db4920 (k-slock-AF_INET6){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:373 [inline]
 #5: ffff888023db4920 (k-slock-AF_INET6){+...}-{2:2}, at: icmpv6_xmit_lock net/ipv6/icmp.c:118 [inline]
 #5: ffff888023db4920 (k-slock-AF_INET6){+...}-{2:2}, at: icmp6_send+0xcb5/0x21e0 net/ipv6/icmp.c:551
 #6: ffffffff8c91f720 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x5/0x30 include/linux/rcupdate.h:311
 #7: ffffffff8c91f780 (rcu_read_lock_bh){....}-{1:2}, at: rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:312
 #8: ffffffff8c91f780 (rcu_read_lock_bh){....}-{1:2}, at: rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:312

stack backtrace:
CPU: 1 PID: 3843 Comm: syz-executor427 Not tainted 5.15.153-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2cb lib/dump_stack.c:106
 print_deadlock_bug kernel/locking/lockdep.c:2946 [inline]
 check_deadlock kernel/locking/lockdep.c:2989 [inline]
 validate_chain+0x46d2/0x5930 kernel/locking/lockdep.c:3775
 __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
 lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:363 [inline]
 __netif_tx_lock include/linux/netdevice.h:4429 [inline]
 __dev_queue_xmit+0x15be/0x3230 net/core/dev.c:4244
 neigh_output include/net/neighbour.h:509 [inline]
 ip6_finish_output2+0xee8/0x14f0 net/ipv6/ip6_output.c:126
 ip6_send_skb+0x12b/0x240 net/ipv6/ip6_output.c:1943
 icmp6_send+0x1732/0x21e0 net/ipv6/icmp.c:630
 __icmpv6_send include/linux/icmpv6.h:28 [inline]
 icmpv6_send include/linux/icmpv6.h:49 [inline]
 ip6_link_failure+0x37/0x4a0 net/ipv6/route.c:2790
 ip_tunnel_xmit+0x1cee/0x27b0 net/ipv4/ip_tunnel.c:843
 __gre_xmit net/ipv4/ip_gre.c:469 [inline]
 erspan_xmit+0xa9c/0x1530 net/ipv4/ip_gre.c:718
 __netdev_start_xmit include/linux/netdevice.h:5019 [inline]
 netdev_start_xmit include/linux/netdevice.h:5033 [inline]
 xmit_one net/core/dev.c:3617 [inline]
 dev_hard_start_xmit+0x298/0x7a0 net/core/dev.c:3633
 sch_direct_xmit+0x2b2/0x5e0 net/sched/sch_generic.c:342
 __dev_xmit_skb net/core/dev.c:3844 [inline]
 __dev_queue_xmit+0x18ee/0x3230 net/core/dev.c:4213
 neigh_output include/net/neighbour.h:509 [inline]
 ip6_finish_output2+0xee8/0x14f0 net/ipv6/ip6_output.c:126
 dst_output include/net/dst.h:443 [inline]
 NF_HOOK include/linux/netfilter.h:302 [inline]
 rawv6_send_hdrinc+0xceb/0x1800 net/ipv6/raw.c:691
 rawv6_sendmsg+0x160e/0x2100 net/ipv6/raw.c:949
 sock_sendmsg_nosec net/socket.c:704 [inline]
 __sock_sendmsg net/socket.c:716 [inline]
 ____sys_sendmsg+0x59e/0x8f0 net/socket.c:2431
 ___sys_sendmsg+0x252/0x2e0 net/socket.c:2485
 __sys_sendmmsg+0x2bf/0x560 net/socket.c:2571
 __do_sys_sendmmsg net/socket.c:2600 [inline]
 __se_sys_sendmmsg net/socket.c:2597 [inline]
 __x64_sys_sendmmsg+0x9c/0xb0 net/socket.c:2597
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x61/0xcb
RIP: 0033:0x7f70bbb52d29
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 d1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fff668332f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f70bbb52d29
RDX: 0000000000000002 RSI: 0000000020001e80 RDI: 0000000000000005
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000100000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 </TASK>

Crashes (12):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/03/28 02:13 linux-5.15.y 9465fef4ae35 120789fd .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __dev_queue_xmit
2024/03/28 00:50 linux-5.15.y 9465fef4ae35 120789fd .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __dev_queue_xmit
2024/03/27 14:59 linux-5.15.y 9465fef4ae35 454571b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __dev_queue_xmit
2024/03/22 13:42 linux-5.15.y b95c01af2113 4b6cdce6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __dev_queue_xmit
2024/04/09 07:30 linux-5.15.y 9465fef4ae35 f3234354 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __dev_queue_xmit
2024/03/15 18:13 linux-5.15.y 574362648507 d615901c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __dev_queue_xmit
2024/02/06 00:25 linux-5.15.y 6139f2a02fe0 6404acf9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __dev_queue_xmit
2024/01/29 11:33 linux-5.15.y 6139f2a02fe0 991a98f4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __dev_queue_xmit
2024/01/23 21:48 linux-5.15.y ddcaf4999061 1e153dc8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __dev_queue_xmit
2024/01/18 13:44 linux-5.15.y ddcaf4999061 239abf84 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __dev_queue_xmit
2024/01/08 20:36 linux-5.15.y 26c690eff0a5 4c0fd4bb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __dev_queue_xmit
2023/12/15 02:10 linux-5.15.y d0fc081c6241 3222d10c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __dev_queue_xmit
* Struck through repros no longer work on HEAD.