possible deadlock in __dev_queue

possible deadlock in __dev_queue_xmit (3)
Status: upstream: reported C repro on 2019/12/03 09:55
Reported-by: syzbot+3b165dac15094065651e@syzkaller.appspotmail.com
First crash: 463d, last: 3d09h

Cause bisection: introduced by (bisect log) :
commit 1a33e10e4a95cb109ff1145098175df3113313ef
Author: Cong Wang <xiyou.wangcong@gmail.com>
Date: Sun May 3 05:22:19 2020 +0000

net: partially revert dynamic lockdep key changes

Crash: possible deadlock in __dev_queue_xmit (log)
Repro: C syz .config

similar bugs (7):
Kernel	Title	Repro	Count	Last	Reported	Patched	Status
android-49	possible deadlock in __dev_queue_xmit		4	623d	692d	0/3	auto-closed as invalid on 2019/10/25 08:36
linux-4.19	possible deadlock in __dev_queue_xmit	C	4	13d	218d	0/1	upstream: reported C repro on 2020/07/31 07:05
android-414	possible deadlock in __dev_queue_xmit		3	682d	693d	0/1	auto-closed as invalid on 2019/10/21 21:31
android-44	possible deadlock in __dev_queue_xmit		14	458d	520d	0/2	auto-closed as invalid on 2020/04/02 07:14
linux-4.14	possible deadlock in __dev_queue_xmit		7	169d	614d	0/1	auto-closed as invalid on 2021/01/16 21:33
upstream	possible deadlock in __dev_queue_xmit		1	781d	781d	0/21	closed as invalid on 2019/03/10 18:51
upstream	possible deadlock in __dev_queue_xmit (2)		2	593d	709d	0/21	auto-closed as invalid on 2019/11/19 09:01

Sample crash report:

============================================
WARNING: possible recursive locking detected
5.8.0-syzkaller #0 Not tainted
--------------------------------------------
syz-executor951/7234 is trying to acquire lock:
ffff8880953ccc98 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
ffff8880953ccc98 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4108 [inline]
ffff8880953ccc98 (_xmit_ETHER#2){+.-.}-{2:2}, at: __dev_queue_xmit+0x215e/0x2d60 net/core/dev.c:4132

but task is already holding lock:
ffff8880968b6898 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
ffff8880968b6898 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4108 [inline]
ffff8880968b6898 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x256/0xbd0 net/sched/sch_generic.c:311

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(_xmit_ETHER#2);
  lock(_xmit_ETHER#2);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

10 locks held by syz-executor951/7234:
 #0: ffffffff89bd62c0 (rcu_read_lock){....}-{1:2}, at: l3mdev_l3_out include/net/l3mdev.h:198 [inline]
 #0: ffffffff89bd62c0 (rcu_read_lock){....}-{1:2}, at: l3mdev_ip6_out include/net/l3mdev.h:219 [inline]
 #0: ffffffff89bd62c0 (rcu_read_lock){....}-{1:2}, at: rawv6_send_hdrinc net/ipv6/raw.c:677 [inline]
 #0: ffffffff89bd62c0 (rcu_read_lock){....}-{1:2}, at: rawv6_sendmsg+0x1dda/0x38f0 net/ipv6/raw.c:944
 #1: ffffffff89bd6260 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #1: ffffffff89bd6260 (rcu_read_lock_bh){....}-{1:2}, at: ip6_finish_output2+0x190/0x17b0 net/ipv6/ip6_output.c:103
 #2: ffffffff89bd6260 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1da/0x2d60 net/core/dev.c:4071
 #3: ffff8880a1fc5258 (&sch->seqlock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:364 [inline]
 #3: ffff8880a1fc5258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:159 [inline]
 #3: ffff8880a1fc5258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run include/net/pkt_sched.h:128 [inline]
 #3: ffff8880a1fc5258 (&sch->seqlock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3752 [inline]
 #3: ffff8880a1fc5258 (&sch->seqlock){+...}-{2:2}, at: __dev_queue_xmit+0x1310/0x2d60 net/core/dev.c:4105
 #4: ffff8880a1fc5148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_resolve_output net/core/neighbour.c:1489 [inline]
 #4: ffff8880a1fc5148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_resolve_output+0x3fe/0x6a0 net/core/neighbour.c:1469
 #5: ffff8880968b6898 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
 #5: ffff8880968b6898 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4108 [inline]
 #5: ffff8880968b6898 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x256/0xbd0 net/sched/sch_generic.c:311
 #6: ffff8880a5dcd820 (k-slock-AF_INET6){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:364 [inline]
 #6: ffff8880a5dcd820 (k-slock-AF_INET6){+...}-{2:2}, at: icmpv6_xmit_lock net/ipv6/icmp.c:117 [inline]
 #6: ffff8880a5dcd820 (k-slock-AF_INET6){+...}-{2:2}, at: icmp6_send+0xe82/0x2670 net/ipv6/icmp.c:538
 #7: ffffffff89bd62c0 (rcu_read_lock){....}-{1:2}, at: icmp6_send+0x145d/0x2670 net/ipv6/icmp.c:598
 #8: ffffffff89bd6260 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #8: ffffffff89bd6260 (rcu_read_lock_bh){....}-{1:2}, at: ip6_finish_output2+0x190/0x17b0 net/ipv6/ip6_output.c:103
 #9: ffffffff89bd6260 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1da/0x2d60 net/core/dev.c:4071

stack backtrace:
CPU: 0 PID: 7234 Comm: syz-executor951 Not tainted 5.8.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x18f/0x20d lib/dump_stack.c:118
 print_deadlock_bug kernel/locking/lockdep.c:2391 [inline]
 check_deadlock kernel/locking/lockdep.c:2432 [inline]
 validate_chain kernel/locking/lockdep.c:3202 [inline]
 __lock_acquire.cold+0x115/0x396 kernel/locking/lockdep.c:4426
 lock_acquire+0x1f1/0xad0 kernel/locking/lockdep.c:5005
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
 spin_lock include/linux/spinlock.h:354 [inline]
 __netif_tx_lock include/linux/netdevice.h:4108 [inline]
 __dev_queue_xmit+0x215e/0x2d60 net/core/dev.c:4132
 neigh_hh_output include/net/neighbour.h:498 [inline]
 neigh_output include/net/neighbour.h:507 [inline]
 ip6_finish_output2+0x7f1/0x17b0 net/ipv6/ip6_output.c:117
 __ip6_finish_output net/ipv6/ip6_output.c:143 [inline]
 __ip6_finish_output+0x447/0xab0 net/ipv6/ip6_output.c:128
 ip6_finish_output+0x34/0x1f0 net/ipv6/ip6_output.c:153
 NF_HOOK_COND include/linux/netfilter.h:290 [inline]
 ip6_output+0x1db/0x520 net/ipv6/ip6_output.c:176
 dst_output include/net/dst.h:443 [inline]
 ip6_local_out+0xaf/0x1a0 net/ipv6/output_core.c:179
 ip6_send_skb+0xb7/0x340 net/ipv6/ip6_output.c:1867
 ip6_push_pending_frames+0xbd/0xe0 net/ipv6/ip6_output.c:1887
 icmpv6_push_pending_frames+0x294/0x470 net/ipv6/icmp.c:304
 icmp6_send+0x1cf6/0x2670 net/ipv6/icmp.c:617
 icmpv6_send include/linux/icmpv6.h:24 [inline]
 ip6_link_failure+0x29/0x510 net/ipv6/route.c:2669
 dst_link_failure include/net/dst.h:426 [inline]
 ip_tunnel_xmit+0x15cc/0x2ac3 net/ipv4/ip_tunnel.c:822
 erspan_xmit+0x1109/0x2760 net/ipv4/ip_gre.c:704
 __netdev_start_xmit include/linux/netdevice.h:4634 [inline]
 netdev_start_xmit include/linux/netdevice.h:4648 [inline]
 xmit_one net/core/dev.c:3561 [inline]
 dev_hard_start_xmit+0x193/0x950 net/core/dev.c:3577
 sch_direct_xmit+0x2e1/0xbd0 net/sched/sch_generic.c:313
 qdisc_restart net/sched/sch_generic.c:376 [inline]
 __qdisc_run+0x4b9/0x1620 net/sched/sch_generic.c:384
 qdisc_run include/net/pkt_sched.h:134 [inline]
 qdisc_run include/net/pkt_sched.h:126 [inline]
 __dev_xmit_skb net/core/dev.c:3752 [inline]
 __dev_queue_xmit+0x1456/0x2d60 net/core/dev.c:4105
 neigh_resolve_output net/core/neighbour.c:1489 [inline]
 neigh_resolve_output+0x3fe/0x6a0 net/core/neighbour.c:1469
 neigh_output include/net/neighbour.h:509 [inline]
 ip6_finish_output2+0x8b6/0x17b0 net/ipv6/ip6_output.c:117
 __ip6_finish_output net/ipv6/ip6_output.c:143 [inline]
 __ip6_finish_output+0x447/0xab0 net/ipv6/ip6_output.c:128
 ip6_finish_output+0x34/0x1f0 net/ipv6/ip6_output.c:153
 NF_HOOK_COND include/linux/netfilter.h:290 [inline]
 ip6_output+0x1db/0x520 net/ipv6/ip6_output.c:176
 dst_output include/net/dst.h:443 [inline]
 NF_HOOK include/linux/netfilter.h:301 [inline]
 rawv6_send_hdrinc net/ipv6/raw.c:687 [inline]
 rawv6_sendmsg+0x2008/0x38f0 net/ipv6/raw.c:944
 inet_sendmsg+0x99/0xe0 net/ipv4/af_inet.c:817
 sock_sendmsg_nosec net/socket.c:651 [inline]
 sock_sendmsg+0xcf/0x120 net/socket.c:671
 sock_write_iter+0x28c/0x3c0 net/socket.c:998
 call_write_iter include/linux/fs.h:1872 [inline]
 new_sync_write+0x422/0x650 fs/read_write.c:503
 vfs_write+0x5ad/0x730 fs/read_write.c:578
 ksys_write+0x1ee/0x250 fs/read_write.c:631
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x443fd9
Code: e8 8c 07 03 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 bb 09 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ffffdbc0058 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000443fd9
RDX: 0000000000000028 RSI: 0000000020000140 RDI: 0000000000000005
RBP: 00007ffffdbc0060 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000f8e3
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000

Fix bisection attempts:
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro
ci-upstream-kasan-gce	2021/03/03 17:51	upstream	f69d02e3	bc15f7db	.config	log	report	syz	C
ci-upstream-kasan-gce	2021/01/13 00:37	upstream	e609571b	bc15f7db	.config	log	report	syz	C
ci-upstream-kasan-gce	2020/11/01 06:32	upstream	c2dc4c07	bc15f7db	.config	log	report	syz	C
ci-upstream-kasan-gce	2020/10/01 11:41	upstream	60e72093	bc15f7db	.config	log	report	syz	C

Crashes (86):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	VM info	Title
ci-upstream-kasan-gce	2020/08/13 00:26	upstream	fb893de3	bc15f7db	.config	log	report	syz	C
ci-upstream-kasan-gce-root	2020/08/02 20:34	upstream	ac3a0c84	63a73341	.config	log	report	syz	C
ci-upstream-kasan-gce-selinux-root	2020/08/01 05:03	upstream	d8b9faec	d895b3be	.config	log	report	syz	C
ci-upstream-kasan-gce-386	2020/08/13 02:20	upstream	fb893de3	bc15f7db	.config	log	report	syz	C
ci-upstream-net-this-kasan-gce	2020/07/31 07:12	net	27a2145d	8df85ed9	.config	log	report	syz	C
ci-upstream-net-kasan-gce	2020/07/31 07:16	net-next	41d707b7	8df85ed9	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/04 22:00	linux-next	01830e6c	80a06902	.config	log	report	syz	C
ci-upstream-kasan-gce-smack-root	2021/01/26 00:46	upstream	f8ad8187	52e37319	.config	log	report			info	possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce-root	2021/01/22 03:09	upstream	9f29bd8b	d4f4eca5	.config	log	report			info	possible deadlock in __dev_queue_xmit
ci-upstream-kasan-gce	2020/11/24 12:06	upstream	d5beb314	1ab681a4	.config	log	report			info
ci-upstream-kasan-gce-root	2020/11/20 12:48	upstream	4d02da97	0767f13f	.config	log	report			info
ci-upstream-kasan-gce	2020/08/21 16:58	upstream	da2968ff	6436ce4b	.config	log	report
ci-upstream-kasan-gce-root	2020/07/30 01:54	upstream	d3590ebf	233283a1	.config	log	report
ci-upstream-kasan-gce	2020/02/15 19:51	upstream	829e6944	5d7b90f1	.config	log	report
ci-upstream-kasan-gce	2019/12/27 21:08	upstream	46cf053e	be5c2c81	.config	log	report
ci-upstream-net-this-kasan-gce	2020/08/09 18:23	net	7c7ab580	70301872	.config	log	report
ci-upstream-bpf-kasan-gce	2020/06/08 08:07	bpf	e7ed83d6	7751efd0	.config	log	report
ci-upstream-net-kasan-gce	2020/12/14 00:14	net-next	13458ffe	b22a7ec3	.config	log	report			info
ci-upstream-net-kasan-gce	2020/12/04 19:40	net-next	55fd59b0	20366b87	.config	log	report			info
ci-upstream-net-kasan-gce	2020/11/11 14:35	net-next	1aa844b9	cca87986	.config	log	report			info
ci-upstream-net-kasan-gce	2020/08/05 08:06	net-next	da795540	80a06902	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/30 16:31	bpf-next	cb8e59cc	a2cdad9d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/27 20:52	bpf-next	cb8e59cc	ffec44b5	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/11 04:15	bpf-next	cb8e59cc	a6f7998d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/11 02:16	bpf-next	cb8e59cc	a6f7998d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/10 23:44	bpf-next	cb8e59cc	a6f7998d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/08 22:54	bpf-next	cb8e59cc	7604bb03	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/07 08:40	bpf-next	cb8e59cc	e6b89e4e	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/07 06:48	bpf-next	cb8e59cc	e6b89e4e	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/06 23:32	bpf-next	cb8e59cc	e6b89e4e	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/05 14:19	bpf-next	cb8e59cc	2420d1bc	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/05 04:06	bpf-next	cb8e59cc	6720fdef	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/04 02:00	bpf-next	e8224bfe	a5ce5de0	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/03 23:56	bpf-next	e8224bfe	a5ce5de0	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/03 09:24	bpf-next	e8224bfe	f3ba1b5b	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/03 05:04	bpf-next	e8224bfe	f3ba1b5b	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/02 23:14	bpf-next	9a25c1df	52fd7b7d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/02 21:35	bpf-next	9a25c1df	52fd7b7d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/02 06:00	bpf-next	d6e6af8f	a0331e89	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/02 04:09	bpf-next	d6e6af8f	a0331e89	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/02 02:32	bpf-next	d6e6af8f	a0331e89	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/01 12:02	bpf-next	551f08b1	a0331e89	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/01 08:44	bpf-next	551f08b1	a0331e89	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/01 06:52	bpf-next	551f08b1	a0331e89	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/01 01:40	bpf-next	551f08b1	a0331e89	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/31 17:57	bpf-next	551f08b1	a0331e89	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/30 03:01	bpf-next	551f08b1	3905eaae	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 01:21	bpf-next	9b185fa4	0d951763	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/28 22:53	bpf-next	9b185fa4	0d951763	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/28 00:17	bpf-next	c41f9b73	ec153193	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/27 21:56	bpf-next	c41f9b73	ec153193	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/27 07:23	bpf-next	c409dc81	9072c126	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/26 23:09	bpf-next	c409dc81	9072c126	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/26 16:10	bpf-next	ff5076b1	8ca3b7d2	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/26 13:08	bpf-next	ff5076b1	8ca3b7d2	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/26 10:25	bpf-next	ff5076b1	8ca3b7d2	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/26 01:50	bpf-next	ff5076b1	30927cd7	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/25 20:40	bpf-next	a152b859	30927cd7	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/25 11:18	bpf-next	a152b859	11284182	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/25 10:42	bpf-next	a152b859	11284182	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/07 05:47	bpf-next	1a323ea5	99a96044	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/02/11 03:02	linux-next	ac431e2d	084454ae	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/02/06 13:42	linux-next	a0c61bf1	c91cbc9d	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2019/11/29 08:53	linux-next	419593da	76357d6f	.config	log	report