syzbot


possible deadlock in __dev_queue_xmit (3)

Status: upstream: reported C repro on 2019/12/03 09:55
Reported-by: syzbot+3b165dac15094065651e@syzkaller.appspotmail.com
First crash: 1032d, last: 1d20h

Cause bisection: introduced by (bisect log) :
commit 1a33e10e4a95cb109ff1145098175df3113313ef
Author: Cong Wang <xiyou.wangcong@gmail.com>
Date: Sun May 3 05:22:19 2020 +0000

  net: partially revert dynamic lockdep key changes

Crash: possible deadlock in __dev_queue_xmit (log)
Repro: C syz .config

Fix bisection: the fix commit could be any of (bisect log):
  fb893de323e2 Merge tag 'tag-chrome-platform-for-v5.9' of git://git.kernel.org/pub/scm/linux/kernel/git/chrome-platform/linux
  ad9f25d33860 Merge tag 'netfs-lib-fixes-20200525' of git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs
similar bugs (7):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
android-49 possible deadlock in __dev_queue_xmit 4 1191d 1261d 0/3 auto-closed as invalid on 2019/10/25 08:36
linux-4.19 possible deadlock in __dev_queue_xmit C error 5 368d 787d 0/1 upstream: reported C repro on 2020/07/31 07:05
android-414 possible deadlock in __dev_queue_xmit 3 1250d 1261d 0/1 auto-closed as invalid on 2019/10/21 21:31
android-44 possible deadlock in __dev_queue_xmit 14 1027d 1088d 0/2 auto-closed as invalid on 2020/04/02 07:14
linux-4.14 possible deadlock in __dev_queue_xmit 7 737d 1182d 0/1 auto-closed as invalid on 2021/01/16 21:33
upstream possible deadlock in __dev_queue_xmit 1 1349d 1349d 0/24 closed as invalid on 2019/03/10 18:51
upstream possible deadlock in __dev_queue_xmit (2) 2 1162d 1277d 0/24 auto-closed as invalid on 2019/11/19 09:01

Sample crash report:
============================================
WARNING: possible recursive locking detected
6.0.0-rc4-syzkaller #0 Not tainted
--------------------------------------------
syz-executor153/3644 is trying to acquire lock:
ffff88801ceba218 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_lock include/linux/spinlock.h:349 [inline]
ffff88801ceba218 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3847 [inline]
ffff88801ceba218 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x2fb8/0x3ad0 net/core/dev.c:4222

but task is already holding lock:
ffff888026882258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:359 [inline]
ffff888026882258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:187 [inline]
ffff888026882258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:184 [inline]
ffff888026882258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3804 [inline]
ffff888026882258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x133c/0x3ad0 net/core/dev.c:4222

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(dev->qdisc_tx_busylock ?: &qdisc_tx_busylock);
  lock(dev->qdisc_tx_busylock ?: &qdisc_tx_busylock);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

5 locks held by syz-executor153/3644:
 #0: ffffffff8bf868a0 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:95 [inline]
 #0: ffffffff8bf868a0 (rcu_read_lock_bh){....}-{1:2}, at: ip_finish_output2+0x292/0x2170 net/ipv4/ip_output.c:214
 #1: ffffffff8bf868a0 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x22f/0x3ad0 net/core/dev.c:4173
 #2: ffff888026882258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:359 [inline]
 #2: ffff888026882258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:187 [inline]
 #2: ffff888026882258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:184 [inline]
 #2: ffff888026882258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3804 [inline]
 #2: ffff888026882258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x133c/0x3ad0 net/core/dev.c:4222
 #3: ffffffff8bf868a0 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:95 [inline]
 #3: ffffffff8bf868a0 (rcu_read_lock_bh){....}-{1:2}, at: ip_finish_output2+0x292/0x2170 net/ipv4/ip_output.c:214
 #4: ffffffff8bf868a0 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x22f/0x3ad0 net/core/dev.c:4173

stack backtrace:
CPU: 1 PID: 3644 Comm: syz-executor153 Not tainted 6.0.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/26/2022
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 print_deadlock_bug kernel/locking/lockdep.c:2988 [inline]
 check_deadlock kernel/locking/lockdep.c:3031 [inline]
 validate_chain kernel/locking/lockdep.c:3816 [inline]
 __lock_acquire.cold+0x116/0x3a7 kernel/locking/lockdep.c:5053
 lock_acquire kernel/locking/lockdep.c:5666 [inline]
 lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5631
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:349 [inline]
 __dev_xmit_skb net/core/dev.c:3847 [inline]
 __dev_queue_xmit+0x2fb8/0x3ad0 net/core/dev.c:4222
 dev_queue_xmit include/linux/netdevice.h:3008 [inline]
 neigh_hh_output include/net/neighbour.h:535 [inline]
 neigh_output include/net/neighbour.h:549 [inline]
 ip_finish_output2+0x14d3/0x2170 net/ipv4/ip_output.c:228
 __ip_finish_output net/ipv4/ip_output.c:306 [inline]
 __ip_finish_output+0x396/0x650 net/ipv4/ip_output.c:288
 ip_finish_output+0x2d/0x280 net/ipv4/ip_output.c:316
 NF_HOOK_COND include/linux/netfilter.h:296 [inline]
 ip_output+0x19f/0x310 net/ipv4/ip_output.c:430
 dst_output include/net/dst.h:451 [inline]
 ip_local_out+0xaf/0x1a0 net/ipv4/ip_output.c:126
 iptunnel_xmit+0x67a/0x9e0 net/ipv4/ip_tunnel_core.c:82
 ip_tunnel_xmit+0x143f/0x3170 net/ipv4/ip_tunnel.c:813
 gre_tap_xmit+0x504/0x630 net/ipv4/ip_gre.c:743
 __netdev_start_xmit include/linux/netdevice.h:4819 [inline]
 netdev_start_xmit include/linux/netdevice.h:4833 [inline]
 xmit_one net/core/dev.c:3590 [inline]
 dev_hard_start_xmit+0x183/0x880 net/core/dev.c:3606
 sch_direct_xmit+0x19f/0xbe0 net/sched/sch_generic.c:342
 __dev_xmit_skb net/core/dev.c:3817 [inline]
 __dev_queue_xmit+0x1568/0x3ad0 net/core/dev.c:4222
 dev_queue_xmit include/linux/netdevice.h:3008 [inline]
 neigh_hh_output include/net/neighbour.h:535 [inline]
 neigh_output include/net/neighbour.h:549 [inline]
 ip_finish_output2+0x14d3/0x2170 net/ipv4/ip_output.c:228
 __ip_finish_output net/ipv4/ip_output.c:306 [inline]
 __ip_finish_output+0x396/0x650 net/ipv4/ip_output.c:288
 ip_finish_output+0x2d/0x280 net/ipv4/ip_output.c:316
 NF_HOOK_COND include/linux/netfilter.h:296 [inline]
 ip_output+0x19f/0x310 net/ipv4/ip_output.c:430
 dst_output include/net/dst.h:451 [inline]
 ip_local_out net/ipv4/ip_output.c:126 [inline]
 ip_send_skb+0xd4/0x260 net/ipv4/ip_output.c:1589
 udp_send_skb+0x725/0x1430 net/ipv4/udp.c:969
 udp_sendmsg+0x1bb8/0x2740 net/ipv4/udp.c:1256
 udpv6_sendmsg+0x17da/0x2c70 net/ipv6/udp.c:1365
 inet6_sendmsg+0x99/0xe0 net/ipv6/af_inet6.c:653
 sock_sendmsg_nosec net/socket.c:714 [inline]
 sock_sendmsg+0xcf/0x120 net/socket.c:734
 ____sys_sendmsg+0x334/0x810 net/socket.c:2482
 ___sys_sendmsg+0x110/0x1b0 net/socket.c:2536
 __sys_sendmmsg+0x18b/0x460 net/socket.c:2622
 __do_sys_sendmmsg net/socket.c:2651 [inline]
 __se_sys_sendmmsg net/socket.c:2648 [inline]
 __x64_sys_sendmmsg+0x99/0x100 net/socket.c:2648
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f30b21b7da9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 41 15 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 c0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffee1d01c98 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 00007ffee1d01cc0 RCX: 00007f30b21b7da9
RDX: 0000000000026700 RSI: 00000000200092c0 RDI: 0000000000000005
RBP: 0000000000000000 R08: 000000000000000d R09: 000000000000000d
R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffee1d01cb0
R13: 00000000000f4240 R14: 000000000000e28b R15: 00007ffee1d01ca4
 </TASK>
IPv6: ADDRCONF(NETDEV_CHANGE): vcan0: link becomes ready
chnl_net:chnl_net_open(): err: Unable to register and open device, Err:-19
caif:caif_disconnect_client(): nothing to disconnect
chnl_net:chnl_flowctrl_cb(): NET flowctrl func called flow: CLOSE/DEINIT
chnl_net:chnl_net_open(): state disconnected

Crashes (672):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-upstream-kasan-gce-selinux-root 2022/09/06 07:54 upstream 7e18e42e4b28 9dcd38fc .config log report syz C possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce-smack-root 2022/08/08 13:50 upstream 200e340f2196 88e3a122 .config log report syz C possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce-selinux-root 2022/07/11 06:05 upstream d9919d43cbf6 b5765a15 .config log report syz C possible deadlock in __dev_queue_xmit
ci-qemu-upstream 2022/05/08 03:29 upstream 30c8e80f7932 e60b1103 .config log report syz C possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce-root 2022/04/22 18:26 upstream d569e86915b7 131df97d .config log report syz C possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce 2022/04/16 11:23 upstream 59250f8a7f3a 8bcc32a6 .config log report syz C possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce 2022/01/04 01:04 upstream c9e6606c7fe9 4a3f34f2 .config log report syz C possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/04/16 15:14 net d08ed852560e 8bcc32a6 .config log report syz C possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/01/04 00:58 net 1ef5e1d0dca5 4a3f34f2 .config log report syz C possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/04/16 10:49 net-next 0339d25a2807 8bcc32a6 .config log report syz C possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/01/03 22:02 net-next 3d694552fd8f 4a3f34f2 .config log report syz C possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce 2020/08/13 00:26 upstream fb893de323e2 bc15f7db .config log report syz C
ci-upstream-kasan-gce-root 2020/08/02 20:34 upstream ac3a0c847296 63a73341 .config log report syz C
ci-upstream-kasan-gce-selinux-root 2020/08/01 05:03 upstream d8b9faec54ae d895b3be .config log report syz C
ci-upstream-kasan-gce-386 2020/08/13 02:20 upstream fb893de323e2 bc15f7db .config log report syz C
ci-upstream-net-this-kasan-gce 2020/07/31 07:12 net 27a2145d6f82 8df85ed9 .config log report syz C
ci-upstream-net-kasan-gce 2020/07/31 07:16 net-next 41d707b7332f 8df85ed9 .config log report syz C
ci-upstream-linux-next-kasan-gce-root 2020/08/04 22:00 linux-next 01830e6c042e 80a06902 .config log report syz C
ci-upstream-kasan-gce-root 2022/09/14 01:13 upstream d1221cea11fc b884348d .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce-smack-root 2022/09/01 03:12 upstream c5e4d5e99162 b01ec571 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce-selinux-root 2022/08/22 18:40 upstream 1c23f9e627a7 26a13b38 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce 2022/08/22 15:42 upstream 1c23f9e627a7 26a13b38 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce 2022/08/15 15:26 upstream 7ebfc85e2cd7 8dfcaa3d .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce 2022/08/12 21:00 upstream 7ebfc85e2cd7 402cd70d .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce-selinux-root 2022/08/09 04:08 upstream 200e340f2196 da700653 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce-root 2022/07/31 10:32 upstream 6a010258447d fef302b1 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce-selinux-root 2022/06/15 09:42 upstream 018ab4fabddd 127d1faf .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce 2022/05/29 18:23 upstream 664a393a2663 a46af346 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce-386 2022/08/05 06:49 upstream 200e340f2196 1c9013ac .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce-386 2022/06/25 01:35 upstream 6a0a17e6c6d1 a371c43c .config log report info possible deadlock in __dev_queue_xmit
ci-qemu-upstream-386 2022/04/27 19:09 upstream 03498b7131b8 8a1f1f07 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/09/24 13:39 net b7ca8d5f56e6 0042f2b4 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/09/22 15:34 net e738455b2c6d 0042f2b4 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/09/20 07:04 net 182447b12144 dd9a85ff .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/09/14 18:01 net 0727a9a5fbc1 b884348d .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/08/29 21:24 net 1bd3a383075c 5b44472d .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/08/29 00:29 net ebe5555c2f34 07177916 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/08/28 06:32 net ebe5555c2f34 07177916 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/08/23 23:58 net 1eec80946a73 cea8b0f7 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/08/12 10:44 net 7ebfc85e2cd7 402cd70d .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/07/27 12:36 net b5177ed92bf6 da9d0366 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/07/21 01:08 net 44484fa8eedf 88cb1383 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/07/18 20:37 net 76c16d3e1944 ff988920 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/07/15 18:40 net 8f3184b95116 95cb00d1 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/07/13 09:37 net 22b9c41a3fb8 5d921b08 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/07/12 11:38 net 4a46de446d3f da3d6955 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/06/30 21:19 net 0a18d802d65c 1434eec0 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/06/29 20:56 net 5a478a653b4c 1434eec0 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/06/23 04:43 net 1e70212e0315 912f5df7 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/05/14 17:29 net 9500acc631db 744a39e2 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/05/11 22:41 net 3cc5c6a7829a beb0b407 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-this-kasan-gce 2022/05/09 18:37 net 49e6123c65da 8b277b8e .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/09/18 14:47 net-next 44a8535fb87c dd9a85ff .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/09/12 20:36 net-next 169ccf0e4082 f371ed7e .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/09/05 02:45 net-next 9837ec955b46 28811d0a .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/07/14 05:09 net-next 6a605eb1d71e 5d921b08 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/07/11 15:42 net-next 0076cad30135 da3d6955 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/07/10 23:42 net-next 0076cad30135 b5765a15 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/07/02 06:48 net-next dbdd9a28e140 1434eec0 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/06/24 02:39 net-next 85763435d5b5 912f5df7 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/06/20 04:57 net-next 9776fe0f424b 8f633d84 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/05/23 13:49 net-next 1e39b27bd9d1 4c7657cb .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/05/23 08:14 net-next 60f243ad1426 7268fa62 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/05/13 01:06 net-next b33177f1d62b 9ad6612a .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/05/11 19:46 net-next 01f4685797a5 beb0b407 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-net-kasan-gce 2022/05/09 09:21 net-next c908565eecf2 e60b1103 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-bpf-next-kasan-gce 2021/08/25 05:38 bpf-next 8c0bb89e8e4d b599f2fc .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-linux-next-kasan-gce-root 2022/08/16 14:06 linux-next 6c8f479764eb 7a7cb304 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-linux-next-kasan-gce-root 2022/06/15 18:49 linux-next 6012273897fe 1719ee24 .config log report info possible deadlock in __dev_queue_xmit
ci-upstream-bpf-kasan-gce 2020/06/08 08:07 bpf e7ed83d6fa1a 7751efd0 .config log report
ci-upstream-net-kasan-gce 2020/12/14 00:14 net-next 13458ffe0a95 b22a7ec3 .config log report info
ci-upstream-linux-next-kasan-gce-root 2019/11/29 08:53 linux-next 419593dad843 76357d6f .config log report
* Struck through repros no longer work on HEAD.