syzbot


possible deadlock in __dev_queue_xmit (2)

Status: auto-closed as invalid on 2019/11/19 09:01
Reported-by: syzbot+f8c40b4da41f3e8049c4@syzkaller.appspotmail.com
First crash: 1234d, last: 1118d
similar bugs (7):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
android-49 possible deadlock in __dev_queue_xmit 4 1148d 1217d 0/3 auto-closed as invalid on 2019/10/25 08:36
upstream possible deadlock in __dev_queue_xmit (3) C done inconclusive 654 1d03h 984d 0/23 upstream: reported C repro on 2019/12/03 09:55
linux-4.19 possible deadlock in __dev_queue_xmit C error 5 325d 743d 0/1 upstream: reported C repro on 2020/07/31 07:05
android-414 possible deadlock in __dev_queue_xmit 3 1207d 1218d 0/1 auto-closed as invalid on 2019/10/21 21:31
android-44 possible deadlock in __dev_queue_xmit 14 983d 1045d 0/2 auto-closed as invalid on 2020/04/02 07:14
linux-4.14 possible deadlock in __dev_queue_xmit 7 694d 1138d 0/1 auto-closed as invalid on 2021/01/16 21:33
upstream possible deadlock in __dev_queue_xmit 1 1306d 1306d 0/23 closed as invalid on 2019/03/10 18:51

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.2.0+ #41 Not tainted
------------------------------------------------------
syz-executor.5/22304 is trying to acquire lock:
00000000d743d280 (_xmit_TUNNEL6#2){+.-.}, at: spin_lock /./include/linux/spinlock.h:338 [inline]
00000000d743d280 (_xmit_TUNNEL6#2){+.-.}, at: __netif_tx_lock /./include/linux/netdevice.h:3890 [inline]
00000000d743d280 (_xmit_TUNNEL6#2){+.-.}, at: __dev_queue_xmit+0x2be1/0x3660 /net/core/dev.c:3865

but task is already holding lock:
000000001fdbeb5c (_xmit_ETHER#2){+.-.}, at: spin_lock /./include/linux/spinlock.h:338 [inline]
000000001fdbeb5c (_xmit_ETHER#2){+.-.}, at: __netif_tx_lock /./include/linux/netdevice.h:3890 [inline]
000000001fdbeb5c (_xmit_ETHER#2){+.-.}, at: sch_direct_xmit+0x2e0/0xc30 /net/sched/sch_generic.c:306

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (_xmit_ETHER#2){+.-.}:
       __raw_spin_lock /./include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2f/0x40 /kernel/locking/spinlock.c:151
       spin_lock /./include/linux/spinlock.h:338 [inline]
       __netif_tx_lock /./include/linux/netdevice.h:3890 [inline]
       sch_direct_xmit+0x2e0/0xc30 /net/sched/sch_generic.c:306
       __dev_xmit_skb /net/core/dev.c:3477 [inline]
       __dev_queue_xmit+0x256b/0x3660 /net/core/dev.c:3838
       dev_queue_xmit+0x18/0x20 /net/core/dev.c:3902
       neigh_resolve_output /net/core/neighbour.c:1488 [inline]
       neigh_resolve_output+0x5a5/0x970 /net/core/neighbour.c:1468
       neigh_output /./include/net/neighbour.h:511 [inline]
       ip6_finish_output2+0x1034/0x2550 /net/ipv6/ip6_output.c:116
       __ip6_finish_output+0x444/0xaa0 /net/ipv6/ip6_output.c:142
       ip6_finish_output+0x38/0x1f0 /net/ipv6/ip6_output.c:152
       NF_HOOK_COND /./include/linux/netfilter.h:294 [inline]
       ip6_output+0x235/0x7f0 /net/ipv6/ip6_output.c:175
       dst_output /./include/net/dst.h:436 [inline]
       NF_HOOK /./include/linux/netfilter.h:305 [inline]
       ndisc_send_skb+0xf29/0x14a0 /net/ipv6/ndisc.c:504
       ndisc_send_ns+0x3a9/0x850 /net/ipv6/ndisc.c:646
       ndisc_solicit+0x2ed/0x470 /net/ipv6/ndisc.c:738
       neigh_probe+0xc6/0x110 /net/core/neighbour.c:1012
       __neigh_event_send+0x3a3/0x1700 /net/core/neighbour.c:1170
       neigh_event_send /./include/net/neighbour.h:445 [inline]
       neigh_resolve_output+0x5cf/0x970 /net/core/neighbour.c:1472
       neigh_output /./include/net/neighbour.h:511 [inline]
       ip6_finish_output2+0x1034/0x2550 /net/ipv6/ip6_output.c:116
       __ip6_finish_output+0x444/0xaa0 /net/ipv6/ip6_output.c:142
       ip6_finish_output+0x38/0x1f0 /net/ipv6/ip6_output.c:152
       NF_HOOK_COND /./include/linux/netfilter.h:294 [inline]
       ip6_output+0x235/0x7f0 /net/ipv6/ip6_output.c:175
       dst_output /./include/net/dst.h:436 [inline]
       ip6_local_out+0xbb/0x1b0 /net/ipv6/output_core.c:179
       ip6_send_skb+0xbb/0x350 /net/ipv6/ip6_output.c:1791
       ip6_push_pending_frames+0xc8/0xf0 /net/ipv6/ip6_output.c:1811
       icmpv6_push_pending_frames+0x34b/0x540 /net/ipv6/icmp.c:285
       icmp6_send+0x1970/0x1e20 /net/ipv6/icmp.c:586
       icmpv6_send+0xec/0x230 /net/ipv6/ip6_icmp.c:43
       ip6_link_failure+0x2b/0x530 /net/ipv6/route.c:2640
       dst_link_failure /./include/net/dst.h:419 [inline]
       vti6_xmit /net/ipv6/ip6_vti.c:510 [inline]
       vti6_tnl_xmit+0x10c4/0x1c00 /net/ipv6/ip6_vti.c:549
       __netdev_start_xmit /./include/linux/netdevice.h:4406 [inline]
       netdev_start_xmit /./include/linux/netdevice.h:4420 [inline]
       xmit_one /net/core/dev.c:3280 [inline]
       dev_hard_start_xmit+0x1a3/0x9c0 /net/core/dev.c:3296
       __dev_queue_xmit+0x2b15/0x3660 /net/core/dev.c:3869
       dev_queue_xmit+0x18/0x20 /net/core/dev.c:3902
       neigh_direct_output+0x16/0x20 /net/core/neighbour.c:1528
       neigh_output /./include/net/neighbour.h:511 [inline]
       ip6_finish_output2+0x1034/0x2550 /net/ipv6/ip6_output.c:116
       __ip6_finish_output+0x444/0xaa0 /net/ipv6/ip6_output.c:142
       ip6_finish_output+0x38/0x1f0 /net/ipv6/ip6_output.c:152
       NF_HOOK_COND /./include/linux/netfilter.h:294 [inline]
       ip6_output+0x235/0x7f0 /net/ipv6/ip6_output.c:175
       dst_output /./include/net/dst.h:436 [inline]
       NF_HOOK /./include/linux/netfilter.h:305 [inline]
       rawv6_send_hdrinc /net/ipv6/raw.c:687 [inline]
       rawv6_sendmsg+0x1f50/0x3700 /net/ipv6/raw.c:942
       inet_sendmsg+0x9e/0xe0 /net/ipv4/af_inet.c:807
       sock_sendmsg_nosec /net/socket.c:633 [inline]
       sock_sendmsg+0xd7/0x130 /net/socket.c:653
       ___sys_sendmsg+0x803/0x920 /net/socket.c:2307
       __sys_sendmsg+0x105/0x1d0 /net/socket.c:2352
       __do_sys_sendmsg /net/socket.c:2361 [inline]
       __se_sys_sendmsg /net/socket.c:2359 [inline]
       __x64_sys_sendmsg+0x78/0xb0 /net/socket.c:2359
       do_syscall_64+0xfd/0x6a0 /arch/x86/entry/common.c:296
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #0 (_xmit_TUNNEL6#2){+.-.}:
       check_prev_add /kernel/locking/lockdep.c:2405 [inline]
       check_prevs_add /kernel/locking/lockdep.c:2507 [inline]
       validate_chain /kernel/locking/lockdep.c:2897 [inline]
       __lock_acquire+0x25a0/0x4690 /kernel/locking/lockdep.c:3880
       lock_acquire+0x190/0x410 /kernel/locking/lockdep.c:4413
       __raw_spin_lock /./include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2f/0x40 /kernel/locking/spinlock.c:151
       spin_lock /./include/linux/spinlock.h:338 [inline]
       __netif_tx_lock /./include/linux/netdevice.h:3890 [inline]
       __dev_queue_xmit+0x2be1/0x3660 /net/core/dev.c:3865
       dev_queue_xmit+0x18/0x20 /net/core/dev.c:3902
       neigh_direct_output+0x16/0x20 /net/core/neighbour.c:1528
       neigh_output /./include/net/neighbour.h:511 [inline]
       ip_finish_output2+0x8a5/0x2590 /net/ipv4/ip_output.c:228
       ip_do_fragment+0x944/0x1cf0 /net/ipv4/ip_output.c:903
       ip_fragment.constprop.0+0x176/0x240 /net/ipv4/ip_output.c:575
       __ip_finish_output /net/ipv4/ip_output.c:306 [inline]
       __ip_finish_output+0x4bd/0xbc0 /net/ipv4/ip_output.c:290
       ip_finish_output+0x38/0x1f0 /net/ipv4/ip_output.c:318
       NF_HOOK_COND /./include/linux/netfilter.h:294 [inline]
       ip_mc_output+0x292/0xf70 /net/ipv4/ip_output.c:417
       dst_output /./include/net/dst.h:436 [inline]
       ip_local_out+0xbb/0x1b0 /net/ipv4/ip_output.c:125
       iptunnel_xmit+0x594/0x9b0 /net/ipv4/ip_tunnel_core.c:78
       ip_tunnel_xmit+0x1032/0x2c38 /net/ipv4/ip_tunnel.c:818
       __gre_xmit+0x5e9/0x9a0 /net/ipv4/ip_gre.c:444
       erspan_xmit+0x912/0x28b0 /net/ipv4/ip_gre.c:679
       __netdev_start_xmit /./include/linux/netdevice.h:4406 [inline]
       netdev_start_xmit /./include/linux/netdevice.h:4420 [inline]
       xmit_one /net/core/dev.c:3280 [inline]
       dev_hard_start_xmit+0x1a3/0x9c0 /net/core/dev.c:3296
       sch_direct_xmit+0x372/0xc30 /net/sched/sch_generic.c:308
       __dev_xmit_skb /net/core/dev.c:3477 [inline]
       __dev_queue_xmit+0x256b/0x3660 /net/core/dev.c:3838
       dev_queue_xmit+0x18/0x20 /net/core/dev.c:3902
       neigh_resolve_output /net/core/neighbour.c:1488 [inline]
       neigh_resolve_output+0x5a5/0x970 /net/core/neighbour.c:1468
       neigh_output /./include/net/neighbour.h:511 [inline]
       ip_finish_output2+0x8a5/0x2590 /net/ipv4/ip_output.c:228
       ip_do_fragment+0x17fd/0x1cf0 /net/ipv4/ip_output.c:849
       ip_fragment.constprop.0+0x176/0x240 /net/ipv4/ip_output.c:575
       __ip_finish_output /net/ipv4/ip_output.c:306 [inline]
       __ip_finish_output+0x4bd/0xbc0 /net/ipv4/ip_output.c:290
       ip_finish_output+0x38/0x1f0 /net/ipv4/ip_output.c:318
       NF_HOOK_COND /./include/linux/netfilter.h:294 [inline]
       ip_mc_output+0x292/0xf70 /net/ipv4/ip_output.c:417
       dst_output /./include/net/dst.h:436 [inline]
       ip_local_out+0xbb/0x1b0 /net/ipv4/ip_output.c:125
       ip_send_skb+0x42/0xf0 /net/ipv4/ip_output.c:1554
       udp_send_skb.isra.0+0x6b2/0x1180 /net/ipv4/udp.c:887
       udp_push_pending_frames+0x5c/0xf0 /net/ipv4/udp.c:915
       udp_sendmsg+0x13dd/0x2830 /net/ipv4/udp.c:1206
       inet_sendmsg+0x9e/0xe0 /net/ipv4/af_inet.c:807
       sock_sendmsg_nosec /net/socket.c:633 [inline]
       sock_sendmsg+0xd7/0x130 /net/socket.c:653
       sock_write_iter+0x27c/0x3e0 /net/socket.c:985
       call_write_iter /./include/linux/fs.h:1870 [inline]
       new_sync_write+0x4d3/0x770 /fs/read_write.c:483
       __vfs_write+0xe1/0x110 /fs/read_write.c:496
       vfs_write+0x20c/0x580 /fs/read_write.c:558
       ksys_write+0x14f/0x290 /fs/read_write.c:611
       __do_sys_write /fs/read_write.c:623 [inline]
       __se_sys_write /fs/read_write.c:620 [inline]
       __x64_sys_write+0x73/0xb0 /fs/read_write.c:620
       do_syscall_64+0xfd/0x6a0 /arch/x86/entry/common.c:296
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(_xmit_ETHER#2);
                               lock(_xmit_TUNNEL6#2);
                               lock(_xmit_ETHER#2);
  lock(_xmit_TUNNEL6#2);

 *** DEADLOCK ***

8 locks held by syz-executor.5/22304:
 #0: 0000000052ffdcb3 (sk_lock-AF_INET){+.+.}, at: lock_sock /./include/net/sock.h:1522 [inline]
 #0: 0000000052ffdcb3 (sk_lock-AF_INET){+.+.}, at: udp_sendmsg+0x7cb/0x2830 /net/ipv4/udp.c:1000
 #1: 00000000e61b3b40 (rcu_read_lock_bh){....}, at: lwtunnel_xmit_redirect /./include/net/lwtunnel.h:92 [inline]
 #1: 00000000e61b3b40 (rcu_read_lock_bh){....}, at: ip_finish_output2+0x2dc/0x2590 /net/ipv4/ip_output.c:214
 #2: 00000000e61b3b40 (rcu_read_lock_bh){....}, at: __dev_queue_xmit+0x20a/0x3660 /net/core/dev.c:3804
 #3: 0000000089ff5eab (&(&sch->seqlock)->rlock){+...}, at: spin_trylock /./include/linux/spinlock.h:348 [inline]
 #3: 0000000089ff5eab (&(&sch->seqlock)->rlock){+...}, at: qdisc_run_begin /./include/net/sch_generic.h:161 [inline]
 #3: 0000000089ff5eab (&(&sch->seqlock)->rlock){+...}, at: __dev_xmit_skb /net/core/dev.c:3474 [inline]
 #3: 0000000089ff5eab (&(&sch->seqlock)->rlock){+...}, at: __dev_queue_xmit+0x2faa/0x3660 /net/core/dev.c:3838
 #4: 00000000a1ec5aa8 (dev->qdisc_running_key ?: &qdisc_running_key){+...}, at: dev_queue_xmit+0x18/0x20 /net/core/dev.c:3902
 #5: 000000001fdbeb5c (_xmit_ETHER#2){+.-.}, at: spin_lock /./include/linux/spinlock.h:338 [inline]
 #5: 000000001fdbeb5c (_xmit_ETHER#2){+.-.}, at: __netif_tx_lock /./include/linux/netdevice.h:3890 [inline]
 #5: 000000001fdbeb5c (_xmit_ETHER#2){+.-.}, at: sch_direct_xmit+0x2e0/0xc30 /net/sched/sch_generic.c:306
 #6: 00000000e61b3b40 (rcu_read_lock_bh){....}, at: lwtunnel_xmit_redirect /./include/net/lwtunnel.h:92 [inline]
 #6: 00000000e61b3b40 (rcu_read_lock_bh){....}, at: ip_finish_output2+0x2dc/0x2590 /net/ipv4/ip_output.c:214
 #7: 00000000e61b3b40 (rcu_read_lock_bh){....}, at: __dev_queue_xmit+0x20a/0x3660 /net/core/dev.c:3804

stack backtrace:
CPU: 1 PID: 22304 Comm: syz-executor.5 Not tainted 5.2.0+ #41
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack /lib/dump_stack.c:77 [inline]
 dump_stack+0x172/0x1f0 /lib/dump_stack.c:113
 print_circular_bug.cold+0x163/0x172 /kernel/locking/lockdep.c:1617
 check_noncircular+0x345/0x3e0 /kernel/locking/lockdep.c:1741
 check_prev_add /kernel/locking/lockdep.c:2405 [inline]
 check_prevs_add /kernel/locking/lockdep.c:2507 [inline]
 validate_chain /kernel/locking/lockdep.c:2897 [inline]
 __lock_acquire+0x25a0/0x4690 /kernel/locking/lockdep.c:3880
 lock_acquire+0x190/0x410 /kernel/locking/lockdep.c:4413
 __raw_spin_lock /./include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2f/0x40 /kernel/locking/spinlock.c:151
 spin_lock /./include/linux/spinlock.h:338 [inline]
 __netif_tx_lock /./include/linux/netdevice.h:3890 [inline]
 __dev_queue_xmit+0x2be1/0x3660 /net/core/dev.c:3865
 dev_queue_xmit+0x18/0x20 /net/core/dev.c:3902
 neigh_direct_output+0x16/0x20 /net/core/neighbour.c:1528
 neigh_output /./include/net/neighbour.h:511 [inline]
 ip_finish_output2+0x8a5/0x2590 /net/ipv4/ip_output.c:228
 ip_do_fragment+0x944/0x1cf0 /net/ipv4/ip_output.c:903
 ip_fragment.constprop.0+0x176/0x240 /net/ipv4/ip_output.c:575
 __ip_finish_output /net/ipv4/ip_output.c:306 [inline]
 __ip_finish_output+0x4bd/0xbc0 /net/ipv4/ip_output.c:290
 ip_finish_output+0x38/0x1f0 /net/ipv4/ip_output.c:318
 NF_HOOK_COND /./include/linux/netfilter.h:294 [inline]
 ip_mc_output+0x292/0xf70 /net/ipv4/ip_output.c:417
 dst_output /./include/net/dst.h:436 [inline]
 ip_local_out+0xbb/0x1b0 /net/ipv4/ip_output.c:125
 iptunnel_xmit+0x594/0x9b0 /net/ipv4/ip_tunnel_core.c:78
 ip_tunnel_xmit+0x1032/0x2c38 /net/ipv4/ip_tunnel.c:818
 __gre_xmit+0x5e9/0x9a0 /net/ipv4/ip_gre.c:444
 erspan_xmit+0x912/0x28b0 /net/ipv4/ip_gre.c:679
 __netdev_start_xmit /./include/linux/netdevice.h:4406 [inline]
 netdev_start_xmit /./include/linux/netdevice.h:4420 [inline]
 xmit_one /net/core/dev.c:3280 [inline]
 dev_hard_start_xmit+0x1a3/0x9c0 /net/core/dev.c:3296
 sch_direct_xmit+0x372/0xc30 /net/sched/sch_generic.c:308
 __dev_xmit_skb /net/core/dev.c:3477 [inline]
 __dev_queue_xmit+0x256b/0x3660 /net/core/dev.c:3838
 dev_queue_xmit+0x18/0x20 /net/core/dev.c:3902
 neigh_resolve_output /net/core/neighbour.c:1488 [inline]
 neigh_resolve_output+0x5a5/0x970 /net/core/neighbour.c:1468
 neigh_output /./include/net/neighbour.h:511 [inline]
 ip_finish_output2+0x8a5/0x2590 /net/ipv4/ip_output.c:228
 ip_do_fragment+0x17fd/0x1cf0 /net/ipv4/ip_output.c:849
 ip_fragment.constprop.0+0x176/0x240 /net/ipv4/ip_output.c:575
 __ip_finish_output /net/ipv4/ip_output.c:306 [inline]
 __ip_finish_output+0x4bd/0xbc0 /net/ipv4/ip_output.c:290
 ip_finish_output+0x38/0x1f0 /net/ipv4/ip_output.c:318
 NF_HOOK_COND /./include/linux/netfilter.h:294 [inline]
 ip_mc_output+0x292/0xf70 /net/ipv4/ip_output.c:417
 dst_output /./include/net/dst.h:436 [inline]
 ip_local_out+0xbb/0x1b0 /net/ipv4/ip_output.c:125
 ip_send_skb+0x42/0xf0 /net/ipv4/ip_output.c:1554
 udp_send_skb.isra.0+0x6b2/0x1180 /net/ipv4/udp.c:887
 udp_push_pending_frames+0x5c/0xf0 /net/ipv4/udp.c:915
 udp_sendmsg+0x13dd/0x2830 /net/ipv4/udp.c:1206
 inet_sendmsg+0x9e/0xe0 /net/ipv4/af_inet.c:807
 sock_sendmsg_nosec /net/socket.c:633 [inline]
 sock_sendmsg+0xd7/0x130 /net/socket.c:653
 sock_write_iter+0x27c/0x3e0 /net/socket.c:985
 call_write_iter /./include/linux/fs.h:1870 [inline]
 new_sync_write+0x4d3/0x770 /fs/read_write.c:483
 __vfs_write+0xe1/0x110 /fs/read_write.c:496
 vfs_write+0x20c/0x580 /fs/read_write.c:558
 ksys_write+0x14f/0x290 /fs/read_write.c:611
 __do_sys_write /fs/read_write.c:623 [inline]
 __se_sys_write /fs/read_write.c:620 [inline]
 __x64_sys_write+0x73/0xb0 /fs/read_write.c:620
 do_syscall_64+0xfd/0x6a0 /arch/x86/entry/common.c:296
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x459819
Code: fd b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 cb b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f05d85aac78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000459819
RDX: 000000000000fdef RSI: 0000000020000280 RDI: 0000000000000005
RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f05d85ab6d4
R13: 00000000004c9796 R14: 00000000004e0ba0 R15: 00000000ffffffff

Crashes (2):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-upstream-bpf-next-kasan-gce 2019/07/22 09:00 bpf-next 192f0f8e9db7 b3c615f5 .config log report
ci-upstream-net-kasan-gce 2019/03/28 12:34 net-next 356d71e00d27 f94f56fe .config log report